Evaluating the Implementation of Digital and In-Person Diabetes Prevention Program in a Large, Integrated Health System: Natural Experiment Study Design

Abstract

Introduction:

Implementation of a Diabetes Prevention Program (DPP) in both in-person and digital health-care settings has been increasing. The purpose of this article is to describe the protocol of a mixed-methods, natural experiment study designed to evaluate the implementation of DPP in a large, integrated health system.

Methods:

Kaiser Permanente Northwest patients who were 19 to 75 years with prediabetes (hemoglobin A1c or glycated hemoglobin, 5.7–6.4) and obesity (body mass index ≥ 30 kg/m²) were invited, via the Kaiser Permanente Northwest patient portal, to participate in the digital (n = 4124) and in-person (n = 2669) DPP during 2016 through 2018. Primary (weight) and secondary (hemoglobin A1c or glycated hemoglobin level) outcome data will be obtained from electronic health records. A cost-effectiveness analysis as well as qualitative interviews with patients (enrolled and not enrolled in the DPP) and stakeholders will be conducted to examine further implementation, acceptability, and sustainability.

Conclusion:

The mixed-methods, natural experiment design we will use to evaluate Kaiser Permanente Northwest’s implementation of the digital and in-person DPP builds on existing evidence related to the effectiveness of these two DPP delivery modes and will contribute new knowledge related to best practices for implementing and sustaining the DPP within large health systems over the long term.

Introduction

The prevalence of prediabetes and obesity among US adults 40 years and older is significant, with more 30% having prediabetes and more 40% having obesity.^1,2 Prediabetes and obesity increase the risk for diabetes, cardiovascular disease, and poor quality of life, and are responsible for substantial health-care costs.³ In fact, the estimated costs of managing diabetes in the US was $327 billion in 2017, a 26% increase since 2012.³ In response to the multilevel burden of prediabetes and obesity, there have been several efforts to prevent diabetes at the population level and reduce health-care costs,^4,5 including national implementation and reimbursement of the successful Diabetes Prevention Program (DPP).

The DPP clinical trial demonstrated conclusively that a lifestyle intervention consisting of nutrition, physical activity, and behavioral counseling was more effective in reducing the incidence of type 2 diabetes and producing clinically significant weight loss than were metformin or placebo.⁶ These findings inspired numerous translational studies across the US, conducted in different settings (eg, primary care, community, churches). Meta-analyses of these studies found modest but clinically meaningful effects of the intervention in real-world settings on weight, hemoglobin A1c (HbA1c) and other cardiometabolic risk factors.^7–10 The National Diabetes Prevention Program was created in 2010 to promote DPP dissemination across the US and currently recognizes both in-person and digital (remotely delivered, primarily internet-based) DPPs that meet their standards and operating procedures.¹¹

Beginning in April 2018, the Centers for Medicare & Medicaid Services made a landmark decision to reimburse clinical and nonclinical settings for providing a DPP to Medicare beneficiaries (ie, Medicare DPP); this coverage is currently for in-person DPP only and not digital DPP.^12,13 The Centers for Medicare & Medicaid Services’ decision to cover a DPP among older adults with prediabetes further catapulted efforts within health-care organizations to address the increasing number of individuals with diabetes receiving care in their facilities. The implementation and effectiveness of DPPs within large health systems—primarily the Veterans Health Administration—has been a focus of recent research.^14–17 However, few studies have examined the sustainability of providing a DPP based on maintenance of the effect (ie, long-term weight change and HbA1c levels), health-care costs, participant experience, and organizational support. In addition, engaging individuals in a DPP and similar lifestyle change interventions remains a challenge, and identifying useful approaches is needed.^18–21 Last, although the effectiveness of an in-person DPP is well-established, prior studies evaluating the effect of a digital DPP identified positive outcomes but had some methodological limitations, such as a single arm pre-/posttest design and participant-reported outcomes.^17,22–28

In 2017, Kaiser Permanente Northwest (KPNW), a large, integrated health system serving Oregon and southwest Washington, began piloting both digital and in-person versions of a DPP for its adult health plan members who were 19 to 75 years old with prediabetes (HbA1c, 5.7–6.4%) and obesity [body mass index (BMI) ≥ 30 kg/m²]. We describe a mixed-methods, natural experiment study design to evaluate this large health system initiative by assessing the effects of both a digital and in-person DPP on change in weight and HbA1c level, as well as sustainability based on cost-effectiveness and patient and health-care stakeholder perspectives. We present the approaches KPNW used to identify, recruit, and enroll patients with obesity at high risk for diabetes into a digital and in-person DPP. Furthermore, we describe the two DPP modalities and the mixed-methods approach used in this natural experiment.

Methods

Setting

KPNW provides comprehensive prepaid health care to more than 600,000 members in Oregon and southwest Washington. All patient contacts within the system and all services referred outside are recorded in a single, comprehensive electronic health record (EHR) (KP HealthConnect, based on Epic^®). Approximately 30% of KPNW members have prediabetes, 12% have diabetes, and more than 40% are obese. From 2017 through 2020, KPNW piloted both digital and in-person versions of the DPP for adult health plan members who had both prediabetes and obesity. The decision to target health plan members with both prediabetes and obesity was based on recent predictive modeling using data from more than 77,000 health plan members, which established that an HbA1c level of 6.3 to 6.4 and a BMI ≥30 were associated with more than a 15% probability of developing diabetes in 2 years.²⁹ There were 2 distinct cohorts recruited by the KPNW health system for participation in the diabetes prevention programs. Cohort 1 included KPNW members who were invited by the health-care system to participate in the digital DPP, which was delivered from April 2017 through April 2018. Cohort 2 included KPNW members who were invited by the health-care system to participate in a group-based in-person DPP program across three waves of recruitment and intervention delivery from October 2017 through February 2020. Each cohort is described in further detail later.

Study Design

We use a mixed-methods natural experiment study design to assess clinical and cost outcomes among the two cohorts. Specifically, we will compare DPP enrollees to nonenrollees (digital DPP and in-person DPP cohorts will be analyzed separately) on changes in weight and HbA1c levels. Randomization was not feasible in this real-world implementation of the DPP; therefore, propensity score adjustment will be used to control for potential confounding. All data to be used in the analyses of clinical and cost outcomes are collected as part of standard health-care administration within the KPNW health-care system. Primary qualitative data are collected from a subset of DPP enrollees and nonenrollees in the 2 cohorts, as well as from health-care system providers and stakeholders. This study was approved by the KPNW Institutional Review Board (protocol no. STUDY00000693).

Eligibility and Interventions

Digital DPP (Cohort 1)

Cohort 1 individuals met the following inclusion criteria as noted in the EHR: 1) current KPNW member 65 to 75 years old, 2) BMI ≥ 30, 3) HbA1c level between 5.7 and 6.4, and 4) no prior diagnosis of diabetes. In addition, patients had to use the KPNW electronic patient portal (approximately 80% of all KPNW members meet this criterion).

For Cohort 1, KPNW partnered with Omada Health to offer their program, a Centers for Disease Control and Prevention (CDC)–recognized¹¹ translation of the DPP lifestyle intervention that is delivered in a digital format.³⁰ The program consists of (virtual) small-group support, health coaching from Centers for Disease Control and Prevention–trained lifestyle coaches, a National DPP-approved curriculum,³¹ and electronic behavioral tracking tools. Participants are grouped with other individuals with similar demographics, geographic proximity, and BMI. Group participants communicate via a private social network and discussions are facilitated by a health coach. The health coach also communicates with participants through private messages or phone calls, and provides feedback on weight loss progress, and food and physical activity logs. Lessons from the DPP curriculum are posted each week, and participants can review these at their own pace. To facilitate self-monitoring, participants also receive a wireless weight scale and pedometer. The 12-month digital DPP program includes a 16-week intensive program and a 36-week maintenance program. Omada Health has several studies demonstrating the effectiveness of the program in producing clinically significant weight loss.²² The program was only offered once to KPNW members at no cost as part of this implementation pilot.

In-person DPP (Cohort 2)

Individuals eligible for in-person DPP included a wider age range of 19 to 75 years and a greater diabetes risk level (HbA1c, 6.0–6.4%) to ensure those at highest risk for diabetes were being targeted given limited enrollment slots, but otherwise had the same criteria as cohort 1.

In Fall 2017, KPNW launched their Preventing Diabetes Program, a 12-month group-based, in-person DPP that closely followed the Centers for Disease Control and Prevention National DPP curriculum.¹¹ The in-person DPP is overseen by the KPNW Health Engagement & Wellness Services, and group sessions were led by KPNW-registered dietitians. Participants attended sessions weekly for the first 6 months and then monthly for the next 6 months free of charge. Group sessions included approximately 20 participants each and were held at two KPNW clinic locations at a variety of times (daytime and evening) on weekdays and weekends so that participants could attend at their preferred time and place. Participants received hard copies of all curriculum materials, including logs to track weight, eating behaviors, and physical activity, and were encouraged to weigh themselves at home weekly. At the weekly meetings, participants recorded their weight, number of food records kept, and minutes of physical activity from the previous week, and shared these with the group facilitator.

Recruitment

Beginning in early 2017, KPNW staff identified eligible members based on their most recent BMI and HbA1c level documented in the EHR. Primary care clinicians were sent an email with a list of their eligible patients and were given 14 business days to opt out if they did not want their patients to be recruited for the DPP. The eligible members whose clinicians did not opt out were sent a secure email message within the KPNW patient portal inviting them to enroll in the DPP (Figures 1 and 2). The invitation message was signed as though it came from the patients’ primary care clinician. The patient portal is used by 80% of KPNW members and is a common venue for patients to schedule appointments, refill prescriptions, get visit summaries and lab results, and communicate with care clinicians.

Figure 1.

Digital Diabetes Prevention Program secure invitation email.

Figure 2.

In-person Diabetes Prevention Program secure invitation email.

Figure 3 presents the timeline for cohort identification, recruitment, and delivery of the DPP at KPNW. A total of 4132 individuals were identified who met the criteria for cohort 1 and were sent secure messages in April 2017, inviting them to enroll in the digital DPP, which was the only program available at the time because KPNW’s in-person DPP did not launch until Fall 2017. Members were instructed to click on the unique web link embedded in the message to enroll in the digital DPP (Figure 1). Enrollment was tracked via a unique code, and Omada Health provided enrollment data back to KPNW. KPNW planned to offer a digital DPP once as part of the implementation pilot and, because of limited resources, allow only 500 enrollment slots.

Figure 3.

Kaiser Permanente Northwest timeline for cohort identification, recruitment, and delivery of the Diabetes Prevention Program (DPP). EHR = electronic health record.

Individuals included in cohort 2 were identified beginning in July 2017 and were recruited in 3 waves: first in August 2017, again in April 2018, and then in December 2018. There were 2669 invitation messages sent for in-person DPP. For each wave, invitation emails were sent in 2 to 3 batches during the recruitment period (Figure 2). Reminder emails were sent 14 days after the initial invitation. Enrollment slots were limited to 100 participants in each wave (300 total). In the invitation, members were instructed to call a KPNW health coach if interested in enrolling. Health coaches explained the program details and expectations, and assessed (via phone conversations) participant readiness. If determined to be ready, members signed up for their preferred class day and time.

Recruitment for the digital and in-person DPP did not overlap, as shown in Figure 3. To reduce potential contamination, cohort 1 patients enrolled in the digital DPP were not invited to sign-up for the in-person DPP after that program started. However, eligible patients in cohort 1 who did not enroll in the digital DPP were invited to participate in the in-person DPP.

Data Collection

Clinical and Cost Data

The primary outcome is change in weight at 12 months for both the digital and in-person DPP cohorts. Secondary outcomes include change in HbA1c level at 12 months for both cohorts and, for the digital DPP cohort, change in weight and HbA1c level at 24 months. We will also conduct a cost-effectiveness analysis for each cohort at 12 months and, for the digital DPP cohort, at 24 months. Data used for these analyses come from the EHR and were collected during routine care. Clinical data entered in the EHR (eg, weight and HbA1c level) up to 12 months prior to DPP invitations/recruitment letters being sent out serve as baseline measures. Follow-up data come from the 14-month (for 12-month outcomes) or 26-month (for 24-month outcomes) period following the date the DPP invitation/recruitment letters were sent out; all available follow-up data will be used in analyses. Health-care use events documented in the EHR (ie, visits, pharmacy dispenses) for the 12-month baseline period described will be used to calculate costs per patient using standard costing algorithms^32,33 and Medicare fee schedules, and will be compared to costs based on use events during the same 2 follow-up periods assessed for 12- and 24-month clinical outcomes. Clinical covariates (described further later) will also be obtained from the EHR. We obtained a waiver of informed consent for these population-based analyses.

Qualitative Data

Interview guides were developed to ensure consistency in data collection during the semistructured interviews conducted with patients, health-care stakeholders, and clinicians.^34–36 From September 2017 to July 2018, a subsample of DPP enrollees and nonenrollees from cohorts 1 and 2, were selected randomly and invited to participate in an hour-long, semistructured phone interview. Interview participants were offered a financial incentive for participation. The interview guide focused on patients’ perception of the methods used to invite them to participate in the DPP, patients’ reasons for participating, barriers and facilitators to enrollment and ongoing participation, patients’ perceived sense of usefulness of program components, and recommendations for improving and sustaining the program. In addition, from June 2018 to August 2018, health-care system leaders and clinicians were invited to participate in hour-long semistructured phone interviews to understand barriers and facilitators to sustaining the DPP within the health-care system. Verbal consent was obtained from all interview participants prior to initiation, and all interviews were audio-recorded with participants’ permission and were transcribed professionally.

Data Analysis

Power Considerations for Primary Outcome Analysis

Power calculations were conducted for pairwise comparisons of each DPP modality (digital and in-person) vs usual care with the following assumptions: 1) Bonferroni-adjusted type 1 error rate of 0.05/2 ≈ 0.025, 2) the observed weight loss or HbA1c level change for DPP enrollees would be the same for both DPP modalities, 3) there would be 2000 KPNW members eligible for each DPP cohort, and 4) DPP enrollment in each cohort would vary from as low as 5% of eligible KPNW members (n = 100) to as high as 20% (n = 400). This range of enrollment is consistent with previous experience offering such programs using similar recruitment methods in smaller, select populations of patients with prediabetes. The observed weight/HbA1c level change is thus a weighted average of the assumed true DPP effect in DPP enrollees and the assumed change for those in usual care.

For weight, we used a true DPP effect of 9.85 lbs^13,30 and a usual care effect of 0.93 lb⁶ as an average of reported effects from the literature. For the standard deviation of weight change, we assumed 1.2 lbs.³⁰ As indicated in Table 1, we have excellent power to detect significant differences even with enrollment rates as low as 5%. For the HbA1c level, we used a true DPP effect of a reduction of 0.24% as an average of reported results^30,37 and 0.0% for patients receiving usual care.³⁷ For the standard deviation of HbA1c level change, we assumed 0.07%.³⁰ As indicated in Table 1, with these assumptions we again have excellent power if enrollment rates are at least 10%.

Table 1.

Power for comparison of each Diabetes Prevention Program modality vs usual care^a

	Observed weight lossc			Observed HbA1c decreased
Enrollmentb	UC	DPP	Power	UC	DPP	Power
	5%	0.93	1.38	> 0.99	0.00	0.01	0.989
10%	0.93	1.82	> 0.99	0.00	0.02	> 0.99
15%	0.93	2.27	> 0.99	0.00	0.04	> 0.99
20%	0.93	2.71	> 0.99	0.00	0.05	> 0.99

^aAssumes 2-sided with Bonferroni-adjusted type 1 error of 0.05/2 and n = 2000 eligible per cohort.

^bProportion of DPP participants who enroll in the program (i.e., 5% = 100 DPP enrollees per cohort).

^cAssumes weight loss of 9.85 lbs for DPP enrollees.

^dAssumes HbA1c decrease of 0.24% for DPP enrollees.

DPP = Diabetes Prevention Program; HbA1c: hemoglobin A1c or glycated hemoglobin; UC = usual care.

Analysis of Clinical Outcomes

We modeled 12- and 24-month weight trajectories using a linear mixed-effects model. Based on visual inspection of a scatterplot of weights, we considered using a piecewise linear spline function with a knot at 6 or 7 months given the “checkmark” phenomenon often seen in behavioral weight loss studies.^38,39 Random effects for the intercept and slope(s) were included in the model to allow for person-specific trends in weight trajectories, and we determined the correlation structure for the random effects based on best model fit in terms of Akaike information criterion and Bayesian information criterion values. We will evaluate as covariates the following a priori-chosen variables: age (modeled continuously), race/ethnicity, gender, minutes of exercise per week (0 minutes, 10–140 minutes, ≥ 150 minutes), Charlson comorbidity index score (0, 1, 2, 3, or more),^40,41 baseline tobacco use (current, former, never), census tract-level proportion with college education or higher, census tract-level median household income, baseline weight and HbA1c level, and metformin use (treated as time-varying as always yes after the first dispense of the medication).

To control further for potential confounding, models will be adjusted for propensity score. We will estimate propensity scores for enrolling in the DPP (digital and in-person) by fitting a logistic regression model, with enrollment status as the outcome and all covariates listed earlier as predictors. To test the robustness of our primary findings, we will conduct propensity score matching as a sensitivity analysis. In addition to the covariates and propensity score, we will also include a 2-way interaction term between time and DPP enrollment status (digital and in-person will be modeled separately) to compare the time trend of weight trajectories between enrollees and nonenrollees. Furthermore, we will include 2-way interactions for time by gender and time by weight at enrollment. Model assumptions will be checked by examining residuals and predicted values cross-sectionally and over time. We will estimate marginal means (95% confidence intervals), model-estimated weights averaged over all covariates in the model (ie, all covariates are held constant at their means) at months 1 through 12. The primary contrasts of interest will be the differences in estimated weight change from time of recruitment between enrollees and nonenrollees at 12 months. Similar analyses will be conducted to compare digital DPP enrollees and nonenrollees on 24-month weight trajectories. HbA1c trajectories for 12 months (both cohorts) and 24 months (digital DPP) will be modeled similarly to weight using a linear mixed-effects model. We plan to include the same covariates as the weight model, but exclude the interactions for time by gender and time by weight at enrollment.

Analysis of Cost Outcomes

We will conduct an economic evaluation over the 12-month follow-up period for both the digital and in-person DPP cohorts as well as over the 24-month period for the digital DPP cohort from the perspective of the health plan, following best practices,⁴² and guided by previous economic analyses of DPP interventions.^43–48 We will compare the health-care cost functions between DPP enrollees and nonenrollees for each cohort, adjusting for baseline differences using a propensity score similar to that used in the clinical evaluation, but including adjustments for baseline costs. Cost data will include medical care and the cost of intervention delivery. Medical care use will be enumerated using the EHR and will include pharmacy, primary and specialty care office visits, inpatient stays, and laboratory tests. Health-care events (eg, visits, medication) will be costed using existing algorithms.³³ We will report differences in medical care costs between DPP enrollees and nonenrollees. We will consider methods appropriate for cost data (eg, right skewness and potentially censored follow-up time), including 2-part models,⁴⁹ bootstrapping,^50,51 and proportional means regression.⁵² Last, we will assess the cost-effectiveness of the DPP using net benefit regression methods,^53,54 and we will estimate the probability of the DPP being cost-effective at various levels of willingness-to-pay per unit of improvement in clinical outcomes.⁴²

Analysis of Qualitative Data

A coding dictionary based on review of transcript content and interview questions with patients invited to each DPP format as well as health system stakeholders will be used by 2 trained coders to establish interrater reliability. Any differences in coding will be resolved through discussion, and coders will meet regularly to discuss and refine coding processes. A qualitative database will be compiled, coded, and analyzed using a qualitative software program (NVivo 12). After all transcripts were coded within the software program, we applied text retrieval and grouping functions on specific codes and combinations of codes for a particular topic, and summarized the issues, agreements, and disagreements in the content for each item (eg, barriers to participation compared across both DPP modalities). This type of theme-focused analysis provides qualitative data that can be integrated with our quantitative findings and improve our breadth of understanding of the DPP program, its implementation options, and effectiveness.

Discussion

This study will address 2 of the most pervasive and potentially costly health conditions in the US: obesity and prediabetes. The mixed-methods natural experiment design we will use to evaluate KPNW’s implementation of a digital and in-person DPP builds on existing evidence related to the effectiveness of these two DPP delivery modes and will contribute new knowledge related to best practices for implementing and sustaining a DPP within large health systems over the long term. Specifically, many translational studies have documented the success of an in-person DPP outside the research setting; however, evidence related to digital DPP effectiveness remains limited as a result of varied design limitations in studies conducted to date^17,22–27 and low methodological quality.²⁸ The digital DPP shows promise in facilitating access to diabetes prevention services for individuals with physical or geographic barriers,⁵⁵ and evidence supporting its clinical benefit could facilitate expanded coverage of this delivery mode for Medicare and Medicaid beneficiaries. We propose a rigorous study design to evaluate the digital DPP, using a longitudinal, usual care comparison cohort and objectively assessed measures to determine longer term (12- and 24-month) clinical and cost outcomes while applying propensity score adjustment.

In addition, the sustainability of both in-person and digital programs is less certain, particularly in the context of a large health system hoping to offer a DPP to potentially tens of thousands of patients.^21,56 Further vetting of the Centers for Medicare & Medicaid Services payment model for DPP by health-care systems is needed.^12,55,57,58 Our cost-effectiveness analysis will determine the impact of a digital and in-person DPP on health-care costs and the return on investment for health-care systems, which could have significant implications for the expansion and sustainability of a DPP.

There are some limitations to this study design. Specifically, enrollment slots for both the digital and in-person DPP are capped at 500 and 300 slots, respectively, because of limited resources, which makes it more challenging to assess program reach. The limited number of slots may have implications for generalizability in terms of enrollee characteristics and study outcomes. In addition, the health-care system restricting recruitment to those with a web-based patient portal account may also limit the generalizability of our findings. Although 80% of KPNW members use our patient portal, based on prior studies, patients who underuse the patient portal or other EHR-based tools tend to be from underserved populations with a high burden of diabetes.^59–62 Furthermore, each DPP modality will be offered at separate times and to cohorts of individuals with slightly different eligibility criteria (ie, ranges for age and HbA1c levels at baseline for inclusion). This prevents the opportunity to do a head-to-head comparison of digital vs in-person on enrollment based on patient preference.

Conclusion

The mixed-methods natural experiment design we will use to evaluate KPNW’s implementation of a digital and in-person DPP will build on existing evidence related to DPP effectiveness across the 2 delivery modes on change in weight and HbA1c level at 12 months, and for digital DPP at 24 months. In addition, the cost-effectiveness analysis will determine the impact of a digital and in-person DPP on return on investment for health-care systems and sustainability of the program. Findings from our evaluation will therefore inform best practices for implementing and sustaining a DPP within large health-care systems.