Impact of Performance-Improvement Strategies on the Clinical Care and Outcomes of Patients With Type 2 Diabetes

In 2004, the American Medical Association (AMA) approved the Performance Improvement (PI) educational process as a strategy for improving patient care.¹ Through this process, clinicians earn AMA Physician’s Recognition Award (PRA) Category 1 Credit^TM by completing three stages:

• Stage A: an initial self-assessment by means of patient chart review

• Stage B: completion of one or more certified educational activities, followed by the development and implementation of a personalized improvement plan to address deficiencies identified in Stage A

• Stage C: completion of a second chart review to analyze the impact of their improvement plan

Recognizing the need to improve the care of patients with type 2 diabetes,^2–4 an AMA-style PI initiative was developed in 2008, and a second, updated activity was developed in 2009 to promote evidence-based care practices.⁵ Results from this initiative demonstrated statistically significant changes in clinician behavior.⁶ Despite this demonstrated change in provider behavior, the effect on outcomes at the patient level remained uncertain.

This study sought to answer four questions related to the clinical effectiveness of PI-based continuing medical education (CME):

1. To what extent did participants who completed all three stages of the PI activity achieve improvements in the health of their patients with type 2 diabetes?

2. How did these improvements compare with improvements achieved by clinicians who participated in traditional CME-certified activities?

3. How did patient improvements achieved by PI completers compare with improvements achieved by clinicians who completed the first two stages but did not participate in the final chart review (Stage C) of the PI process?

4. Did PI completers differ from other clinicians in the provision of evidence-based care provided to patients with type 2 diabetes before participation in the PI CME?

Understanding the unique clinical impact of PI CME will allow for the development of improved educational activities for clinicians and ultimately better patient outcomes as a result of clinician participation.

Effect of PI CME on patient health (questions 1–3)

Patient-level clinical data were collected retrospectively for three groups of U.S. clinicians who participated in activities launched in 2008 and 2009:

• PI completers: clinicians who completed Stages A, B, and C of a PI initiative

• PI partial completers: clinicians who completed only Stages A and B of a PI initiative

• Traditional CME participants: clinicians who completed a traditional CME activity designed to enhance PI education, but who did not participate in a PI initiative

All clinicians provided chart-review data from both the pre- and post-intervention periods for 10 patients with type 2 diabetes. Data collected for this study were independent of chart data collected during the PI CME activity that occurred before the design of the research study. A small honorarium was distributed after clinicians submitted 10 patient forms as an incentive for participation in this study. At no time during the study were participants compensated with AMA PRA Category 1 Credit™.

Patient inclusion criteria were as follows: established patients with type 2 diabetes who were seen at least twice in each of the pre- and post-activity periods and whose A1C level was above their individual goal at at least one pre- and one post-activity visit. Patients who were pregnant, < 18 years of age, or > 75 years of age at any visit were excluded. The rationale for the A1C criterion was to evaluate changes in clinician behavior (data not shown).

Before data collection, an independent institutional review board (Chesapeake IRB, Inc., Columbia, Md.) deemed the study to be exempt from oversight because clinicians submitted data without patient identifiers.⁷

Clinician participants were provided the date on which they registered for their respective CME activity and were asked to provide patient data up to 1 year before (i.e., pre-activity) and 1 year after (i.e., post-activity) this date. This date range was modified for the PI completer group after data were submitted because it was determined that the post-activity date range overlapped with Stage A participation and was therefore too early to capture meaningful post-activity data. Participation in the partial completer group was too low to allow for a similar adjustment. For traditional CME participants, pre-activity visits were those occurring up to 1 year before activity registration, and post-activity visits were those occurring up to 1 year after activity registration. Patient visits falling outside of the defined pre- or post-activity period were excluded from analysis.

Clinicians were instructed to submit clinical measurements for A1C, blood pressure, LDL cholesterol, and HDL cholesterol from at least two and up to four diabetes-related visits within each of the pre- and post-activity periods. For patients with multiple visits, the earliest valid pre-activity measure was compared to the latest valid post-activity measure.

Mean A1C, LDL cholesterol, and HDL cholesterol levels for patients in each participant group were calculated in the pre- and post-activity periods. Blood pressure, A1C, and LDL cholesterol levels were grouped categorically as follows: blood pressure < 130/80 or ≥ 130/80 mmHg; A1C < 7, 7–7.5, 7.6–9.0, or > 9.0%; and LDL cholesterol < 100 or ≥100 mg/dl.

Multi-level models incorporating random effects at the patient and provider levels were estimated to compare patient outcomes and participant practices between the pre- and post-activity periods and between participant groups. Linear models were estimated for A1C, LDL cholesterol, and HDL cholesterol. Logistic regression models were estimated for categories of blood pressure and LDL cholesterol levels. An ordinal logistic regression model was estimated for categorical A1C levels. Key comparisons were tested for statistical significance, including differences between PI completers and traditional CME participants and between PI completers and PI partial completers and the amount of change from pre- to post-activity measured between groups.

Baseline performance of PI completers compared to partial completers (question 4)

The Diabetes PI 2009 activity included a required self-assessment questionnaire in addition to the submission of patient chart data. The questionnaire evaluated clinician-reported practice patterns related to general diabetes care, prevention and detection of diabetes-related complications, and glycemic control. These data were used to evaluate the similarity of PI completers to other practitioners who completed the initial chart review in Stage A, but who did not continue any further with the program (i.e., non-completers).

Multi-level logistic regression models incorporating random effects at the provider level were estimated to compare participant self-assessment practices between the PI completers and non-completers. The results of all statistical tests were considered significant if the P value was < 0.05. Non-completers were also surveyed regarding their PI participation experience.

Improvement in clinical outcomes of patients cared for by PI completers (question 1)

One hundred twenty-five past PI participants were eligible for inclusion in this study, and 47 of these participants (38%) completed the required chart review extraction forms. After review, 248 patient charts from 44 participants (35%) met the study inclusion criteria.

A statistically significant improvement was observed in the percentage of patients achieving an A1C < 7%, a blood pressure < 130/80 mmHg, and an LDL cholesterol level < 100 mg/dl after PI activity completion (Figure 1). Statistically significant improvements from the pre- to the post-activity periods were also observed for mean A1C, LDL cholesterol, and HDL cholesterol values (Table 1). The improvement in this latter percentage was especially dramatic, doubling between the two periods.

Figure 1.

Percentage of patients with type 2 diabetes with A1C < 7% (n = 248), blood pressure >130/80 mmHg (n = 248), and LDL cholesterol < 100 mg/dl (n = 207) treated by PI completers before and after PI activity participation.

Table 1.

Values for Type 2 Diabetes Patients of PI Completers and Traditional CME Participants

PI completers compared to traditional CME participants (question 2)

Three hundred thirteen U.S.-based participants completed one of the available traditional CME activities (one of two webcasts or one electronic publication) designed to complement the PI activity but did not participate in the PI activity. Of these, 28 clinicians submitted chart-review data; 225 chart-review extraction forms from 27 clinicians met the study criteria.

Patients treated by traditional CME participants demonstrated statistically significant improvements in all measured values except blood pressure (Table 1). A comparison of patient clinical indicators between PI completers and traditional CME participants showed statistically significant differences in categorical A1C values that favored PI completers. At study completion, PI completers had a greater percentage of patients with A1C levels < 7% and fewer patients with A1C levels between 7.6 and 9.0% compared to traditional CME participants. Changes in the percentage of patients below goal for blood pressure, mean A1C, mean (or distribution of) LDL cholesterol, and mean HDL cholesterol values were not significantly different statistically between these participant groups.

PI completers compared to PI partial completers (question 3)

Forty participants who completed the initial self-assessment stage of a PI activity and submitted an improvement plan but who did not complete the final round of the self-assessment (Stage C) were eligible for evaluation. Of these partial PI completers, 9 (23%) submitted data forms; 65 chart-review extraction forms met the study criteria.

Patients of PI completers experienced statistically significant changes in all clinical indicators from the pre- to the post-activity period (Table 2). Patients treated by PI partial completers similarly demonstrated statistically significant improvements in all clinical values, with the exception of blood pressure, and no statistically significant differences in patient improvements achieved over time were found between the groups.

Table 2.

Values for Type 2 Diabetes Patients of PI Completers and Partial Completers

Characteristics of PI completers compared to non-completers (question 4)

One hundred fifty PI completers from the Diabetes PI 2009 activity were compared to 71 participants who completed only the Stage A chart review. Initial self-assessment data revealed similar demographic and practice characteristics, with the exception that, among participants with a staff member dedicated to patient education, PI completers (n = 71) were more likely than were non-completers (n = 28) to employ a certified diabetes educator (CDE) (61 vs. 36%, P = 0.028) (Supplemental Table 1S, available in the online version of this article from http://clinical.diabetesjournals.org). The two clinical groups were also very similar in the care provided to patients. The only significant difference identified was that PI completers were more likely to discuss smoking cessation with patients than were non-completers (Table 3).

Table 3.

Type 2 Diabetes Patient Care Measures Performed by PI Completers and Non-Completers

Discussion

Audit and feedback is a well-known method for improving practice behaviors. However, PI CME as a formal educational process is relatively new, and its effects are not well understood. It is our belief that this was one of the first studies providing clinical evidence that strongly supports a positive relationship between clinician participation in PI CME and patient outcomes.

Previous analyses of the initial 2008 Diabetes PI initiative by this group found significant improvements in measures of process change but only mild improvements in patient health as measured by glycemic control (A1C).⁶ However, that activity did not mandate data from the same patient in the pre- and post-activity periods and therefore did not allow for an accurate assessment of patient health changes over time.

This study demonstrated that patients with diabetes who are cared for by clinicians who complete all three stages of the PI CME initiative experienced significant improvements in clinical measures of patient health (A1C, blood pressure, LDL cholesterol, and HDL cholesterol). Importantly, the patients of traditional CME participants also demonstrated measurable improvements in A1C, LDL cholesterol, and HDL cholesterol; however, categorical improvements in A1C levels were significantly greater for the patients of clinicians who completed the entire PI initiative than for those of traditional CME participants. Patients whose clinicians completed most, but not all, of the PI CME activity also showed significant changes in these clinical measures; however, clinical improvements in patients of the partial PI completer group were similar to those in patients of the PI completer group. Although our sample size was limited for this study component, the data suggest that the second chart review within the PI educational process may have less influence on patient outcomes than the initial chart review and development of an implementation plan.

Overall, PI completers were similar to their peers, with few exceptions. Compared to non-completers, PI completers discussed smoking cessation more often and were more likely to have a CDE as part of their clinic staff. These results and the finding of similar changes in patient health outcomes between the completer and non-completer groups suggest that clinicians who participate in the majority of the PI activity have the potential to achieve similar improvements in patient health.

We also considered how the patient outcomes in this study compared with observations from a broader, national dataset. Data collected from the most recent Healthcare Effectiveness Data and Information Set (HEDIS)⁸ indicate that, on average, 39, 33, and 45% of patients met the American Diabetes Association–recommended goals of A1C < 7%, blood pressure < 130/80 mmHg, and LDL cholesterol < 100 mg/dl, respectively.⁹ In comparison to our study, a smaller percentage of patients cared for by PI completers met each of these goals before activity participation, and a higher percentage of patients achieved these goals after activity completion. Importantly, after participation, PI completers had fewer patients with an A1C > 9% compared to HEDIS data (12 vs. 32%).

Participation and attrition rates are an inherent problem of PI CME activities.^6,10 Non-completers noted that the extensive time and resource commitment were barriers to their participation. However, data from this study showed gains in clinical health outcomes for patients of PI CME participants, and these gains may be greater than those resulting from traditional CME activities. This suggests that it may be more beneficial for clinicians to focus their educational time and efforts on PI-based activities.

This study had several limitations. First, the time frames during which patient improvements occurred likely varied widely among participants. Although PI participants are encouraged to implement their improvement efforts for 3 months before completing the second chart review, the PI system allows participants to enter data at any time. Thus, the actual time frames differed among participants. Second, patients were required to have an above-goal A1C value to ensure that changes in participant behavior could be evaluated. This requirement may have excluded participants who had patients with well-controlled diabetes and contributed to an underestimation of improvement in glycemic control. Third, chart reviews were retrospective, and bias may have occurred with self-reported actions, including the Hawthorne effect among PI completers. Data were collected over multiple time points to reduce this possibility. Finally, the small sample sizes limited the strength of some of the conclusions that were drawn from this study.

Overall, this study provides a detailed examination of the impact of several of the components of PI CME on patient health. Completion of all three stages of PI CME appears to provide categorical improvements in A1C compared to participation in more traditional CME activities. These findings suggest that self-assessment, improvement planning, implementation based on review of one’s own data, and reassessment of the success of the improvement plan contribute to improvements in patient health.

Although participation in the PI process does not appear to provide additional clinical benefit, the overall more intensive process of PI relative to traditional CME efforts appears to have an important impact on patient health. As the health care system has shifted toward a more performance-reimbursement model, the focus on the quality of clinician performance has become increasingly important. PI CME provides a focused, time-intensive, but effective educational endeavor that may help clinicians achieve performance goals and improve patient health.