Simulated Physician Learning Intervention to Improve Safety and Quality of Diabetes Care: A Randomized Trial

Abstract

OBJECTIVE

To assess two physician learning interventions designed to improve safety and quality of diabetes care delivered by primary care physicians (PCPs).

RESEARCH DESIGN AND METHODS

This group randomized clinical trial included 57 consenting PCPs and their 2,020 eligible adult patients with diabetes. Physicians were randomized to no intervention (group A), a simulated case-based physician learning intervention (group B), or the same simulated case-based learning intervention with physician opinion leader feedback (group C). Dependent variables included A1C values, LDL cholesterol values, pharmacotherapy intensification rates in patients not at clinical goals, and risky prescribing events.

RESULTS

Groups B and C had substantial reductions in risky prescribing of metformin in patients with renal impairment (P = 0.03). Compared with groups A and C, physicians in group B achieved slightly better glycemic control (P = 0.04), but physician intensification of oral glucose-lowering medications was not affected by interventions (P = 0.41). Lipid management improved over time (P < 0.001) but did not differ across study groups (P = 0.67).

CONCLUSIONS

A simulated, case-based learning intervention for physicians significantly reduced risky prescribing events and marginally improved glycemic control in actual patients. The addition of opinion leader feedback did not improve the learning intervention. Refinement and further development of this approach is warranted.

Despite recent improvements, the quality of diabetes care in the U.S. is far from what is recommended based on current scientific evidence (1). Moreover, recent studies (2–4) document frequent risky prescribing events in outpatient diabetes care. Many health care professionals, policy makers, and others view diabetes care improvement and patient safety as an important national priority (5,6).

Diabetes care quality varies widely across physicians, but factors that contribute to this variation are poorly understood (7). Qualitative studies indicate that physicians are aware of diabetes care standards. However, physicians frequently fail to intensify therapy for patients not at evidence-based clinical goals and often use medications in potentially risky ways (8,9). Past work (4) confirms that physicians often fail to 1) set appropriate clinical goals, 2) initiate appropriate pharmacotherapy in a timely fashion, and 3) titrate pharmacotherapy until clinical goals are achieved. Variation in care at the physician level within a single medical group supports the hypothesis that change in physician behavior is a necessary component of care improvement and may add benefit even in medical groups with advanced clinical decision support and other office care systems (10,11).

The difficulty of changing physician behavior is widely assumed (12). Information-based continuing medical education has been shown to be largely ineffective as a means of changing behavior of physicians in office practice. The use of opinion leaders, audit and feedback, and financial incentives are more promising. However, each of these strategies has significant drawbacks or limitations. Opinion leader interventions are time consuming and expensive and may be difficult to replicate because of unique properties of opinion leaders and practice environments (13). Audit and feedback have limited effectiveness, consume a great deal of effort and time, and may not be well suited to tailoring care recommendations to specific needs of individual patients (14). Financial incentives are promising but are expensive, somewhat controversial, and difficult to implement over a wide range of clinical domains.

Simulated, case-based learning tools offer a potentially effective approach to physician behavior change. Theory suggests that such an approach can be effective, especially for complex tasks (15,16). However, most simulated learning approaches make little use of either 1) interactive interfaces that simulate patient response to physician management over a series of clinical encounters or 2) feedback tailored to observed physician treatment patterns. The application of such technology to change physician behavior is in its infancy (17).

This study tests an innovative learning intervention designed to change physician behavior and improve safety and quality of diabetes care. The simulated learning intervention was tested alone and in combination with opinion leader feedback. The intervention used a familiar electronic medical record interface and allowed physicians to iteratively manage a set of simulated patient cases and receive electronic feedback on prescribing behavior over a series of clinical encounters. The intervention was inexpensive, portable, and took about 1 h of physician time.

RESEARCH DESIGN AND METHODS

A group randomized trial was designed to test the hypothesis that primary care physician (PCP) participation in either of two study interventions would improve safety or quality of care delivered to actual patients with diabetes, relative to care delivered by control group PCPs. Four principal dependent variables were measured in actual patients: 1) change in A1C and LDL cholesterol testing rates, 2) change in A1C and LDL cholesterol levels, 3) rates of intensification of glucose or lipid medication when patients are not achieving recommended clinical goals, and 4) change in risky prescribing events related to glycemic control.

Study site and study subjects

The study was conducted at HealthPartners Medical Group, an 18-clinic multispecialty group that provides care to 8,000 adults with diabetes. At the end of 2001, the mean A1C in the 90% of diabetic patients who had an A1C test within a year was 7.2%, and the mean LDL cholesterol was 110 mg/dl, reflecting relatively good baseline diabetes care (11). Approximately 10% of adults with diabetes consulted an endocrinologist each year, most often for one visit (18). Additional information about the study site and antecedent efforts to improve diabetes care can be found in previous publications (11).

Sampling frame and recruitment of PCPs

A sample of 122 potentially eligible PCPs was identified. Physicians in the sample were employed at least 20 h a week at HealthPartners Medical Group as of 1 January 2000 and provided care to a minimum of 10 adult diabetic patients in 2000. Participating PCPs received compensation of $100 for group A, $200 for group B, or $600 for group C, predicated on the differential time commitment to each intervention. Before randomization, 67 consenting physicians were blocked into groups of three based on 1) same specialty (family medicine or internal medicine) and 2) whether they provided care to <50 vs. 50 or more diabetic patients. The intervention was done in early 2002, and patient data to assess impact of the intervention included data from 1 January 2001 to 31 December 2003.

Sampling frame and identification of adults with diabetes

Diabetic patients who received care from study physicians were selected for analysis if they met these criteria: 1) were enrolled in HealthPartners Medical Group on 1 January 2000, 2) were at least 19 years of age on 1 January 2000, or 3) had diagnosed diabetes in 2000 based on meeting at least one of these criteria: two or more outpatient ICD-9 diagnostic codes 250.xx for diabetes or filled one or more prescriptions for a diabetes-specific drug. This diabetes identification method has estimated sensitivity 0.91, specificity 0.99, and positive predictive value 0.94 (19). To be included in the analysis, eligible patients had to have the same study-participating PCP for 2 years before and 1 year after the intervention. Analysis included 2,020 eligible adult diabetic patients.

Description of interventions: control group (group A)

PCPs randomized to this group completed baseline surveys but received no intervention.

Learning intervention (group B)

Using software developed in previous work (17,20), we provided each PCP in group B with three clinical scenarios in the same fixed order. An electronic medical record–like interface permitted multiple virtual patient-physician encounters with each case in the presence of a research assistant with no clinical training. At each simulated patient encounter, the PCP viewed history and physical exam data, recorded impressions, and took a series of actions that were not scripted and could include ordering tests, making referrals to specialists and educators, recommending diet and physical activity, and initiating or titrating various medications for glucose, blood pressure, lipids, or depression.

Actions could be taken at each scheduled visit or phone contact. Follow-up was scheduled at any interval recommended by the physician. At the next encounter, the patient's clinical data (A1C, LDL cholesterol, and blood pressure levels and other data) reflected the effect of actions taken in previous encounters, attenuated by the time-dependent effects of medications, lifestyle recommendations, and recognition and treatment of depression. At each follow-up encounter with the simulated patient, the physician received “learning by doing” feedback in the form of patient responses to actions taken in previous encounters. Each physician dealt with three simulated cases over 60 min; each case was treated for a series of simulated encounters over variable lengths of simulated calendar time. At the end of the three cases, each physician received a printed feedback record of the actions they had taken in each case compared with actions taken by an expert physician who performed the same cases. A more complete description of this intervention software is available (17).

The patient cases seen by each physician were initialized for three important clinical situations: 1) a newly diagnosed type 2 diabetic patient on no medications, 2) a patient with contraindications to insulin sensitizers (metformin and thiazolidinediones) who required insulin initiation and subsequent titration, and 3) a depressed individual with resulting low adherence who required insulin titration.

Simulated cases plus physician opinion leader intervention (group C)

Physicians in group C received the same three simulated cases as group B, the same “learning by doing” feedback based on actions taken, and the same printed feedback summary of actions taken compared with those of an expert physician. In addition, upon completion of the three cases, group C physicians received 60 min of verbal interaction and feedback from a physician opinion leader who observed the physician while he/she performed the simulations and used a predesigned checklist at the completion of the three cases as a tool to discuss potentially problematic treatment issues, as well as to give positive reinforcement for good practice patterns that were observed. Therefore, opinion leader feedback included both positive and negative aspects of physician performance.

Dependent variables

A1C and LDL cholesterol test dates were obtained from electronic clinical databases and were used to assess physician adherence to recommended frequency of these tests in adults with diabetes.

Glycemic and lipid control were assessed by comparing the last preintervention A1C and LDL cholesterol with the last A1C and LDL cholesterol value in the 12-month postintervention period. All A1C assays were performed for routine care at a single, centralized, accredited clinical chemistry laboratory. No changes in assay procedures occurred during the study. Details of the A1C assay (21) (normal range 4.5–6.1%; coefficient of variation 0.058% at A1C 8.8%) and LDL cholesterol assay (22) (calculated only after a 12-h fast and for triglycerides <400 mg/dl) are described elsewhere (11).

Medication moves (including initiation or titration of glucose or lipid drugs) were determined in the 12-month postintervention period for all diabetic patients with a baseline A1C ≥7% or LDL cholesterol ≥100 mg/dl. Medication moves were identified based on drug doses and prescription fills during defined periods of time using pharmacy claims data. Because insulin is often discarded monthly due to expiration, insulin claims data do not accurately reflect actual usage; therefore, insulin dose titration was excluded from the glucose move variable.

The learning intervention focused on one risky prescribing event: use of metformin in patients with serum creatinine values ≥1.5 mg/dl. By design, the learning intervention did not address risky use of metformin in patients with congestive heart failure (CHF).

Independent variables

Analysis was limited to two levels (patients nested within physicians) because intraclass correlation coefficients at the clinic level were small, consistent with previously reported data. Patient-level independent variables obtained from electronic databases included age, sex, and baseline comorbidity score (23). Physician-level independent variables obtained from electronic databases included age, sex, specialty, years of experience, and number of diabetic patients.

Analytic approach

This group randomized trial was analyzed as a nested (patients nested within physicians) cohort pretest-posttest control group design when the variables being examined were available both pre- and postintervention (e.g., A1C value). In this analysis, the coefficient and P value for the time-by-condition interaction term tested the intervention effect by examining differences in condition (groups A, B, and C) and change over time. In some instances, post hoc contrasts were utilized to compare change over time in specific combinations of conditions (24). Nested analysis of postintervention data with regression adjustment for patient age, sex, and comorbidity score was used when examining variables measured only postintervention (e.g., medication moves). In this type of analysis, the intervention effect was tested by the coefficient and P value for condition. This analytic approach was utilized as an alternative method to the time-by-condition analysis in order to assess the effect of study arm after control for patient covariates. Since results using this method were similar to those obtained in the time-by-condition analysis, we report only the time-by-condition results.

Mixed-effects models were estimated with SAS Proc Mixed. For dichotomous outcomes (e.g., test rates), Proc Mixed was also used with and without the Glimmix macro. Because results were similar with and without this macro, we report the more readily interpretable (24) results from models without the Glimmix macro.

A priori sample size calculations assumed 20 providers with 30 diabetic patients each (600 diabetic patients per study arm) would be available for analysis. Effective patient sample size was estimated as n = 311 per arm due to clustering using a measured intraclass correlation coefficient = 0.032 based on eligible physicians. This study was designed with 80% power to detect an A1C difference of 0.3% between study arms, with a two-tailed α = 0.05. This study was reviewed, approved in advance, and monitored by the HealthPartners Institutional Review Board, project no. 00-016. Informed consent was obtained from all physician study subjects. This study is registered as no. NCT00262704 at www.clinicaltrials.org.

RESULTS

We were able to recruit >50% of eligible PCPs to participate in the study. Physician attrition occurred due to noncompletion of the physician baseline survey (n = 1), withdrawal from the study (n = 1), noncompletion of the intervention (n = 3), and departure from the medical group during the study period (n = 3). Two physicians were excluded from analysis because they had insufficient postintervention diabetic patients available due to changes in job responsibilities. Attrition occurred evenly across randomized groups, and final analysis included 19 physicians in each group. Over 97% of those who completed the learning intervention rated their satisfaction with the interventions as excellent or very good after completing the simulated cases.

Table 1 characterizes patients and physicians in each of the three study arms. Randomization at the physician level resulted in similar patient samples except that patients of physicians in group B more often had coronary artery disease and higher Charlson scores. Physician attributes did not differ by group.

Table 1.

Description of patients and PCPs

	Intervention groups				P
	All	A	B	C
Patients (n)	2,020	691	725	604
Age as of intervention date (means ± SD)	64.0 ± 13.0	64.2 ± 12.8	63.4 ± 13.2	64.5 ± 13.1	0.24
Age >65 years (%)	48.6	49.4	45.7	51.2	0.11
Female (%)	42.0	44.6	40.4	41.1	0.24
Coronary heart disease (in 12 months preintervention) (%)	14.0	11.9	16.7	13.3	0.03
Hypertension (in 12 months preintervention) (%)	60.2	61.4	59.6	59.4	0.72
Charlson score (in 12 months preintervention) (means ± SD)	2.0 ± 1.6	1.9 ± 1.5	2.1 ± 1.6	2.0 ± 1.7	0.04
Depression diagnosis (in 12 months preintervention) (%)	7.5	7.4	8.1	7.0	0.71
Preintervention A1C (median, last value in 12 months preintervention)	7.20	7.10	7.30	7.10	0.09
Postintervention A1C (median, last value in 12 months preintervention)	7.20	7.20	7.20	7.30	0.79
Preintervention LDL cholesterol (median, last value in 12 monthspreintervention)	102	102	103	101	0.76
Postintervention LDL cholesterol (median, last value in 12 months preintervention)	100	100	100	99	0.99
Physicians (n)	57	19	19	19
Female (%)	26	37	26	16	0.34
Physician age (mean)	48.5	49.6	47.6	48.0	0.76
Years since graduation (mean)	21.7	22.7	20.8	21.5	0.81
≥80% time spent on patient care (%)	75	77	75	72	0.96
Family practitioners	44%	47%	42%	42%	0.93
Number of eligible diabetes patients (means ± SD)	35.4 ± 21.2	36.3 ± 23.0	38.2 ± 22.0	31.8 ± 19.1	0.64

Table 2 shows changes in A1C testing and A1C levels. There was a significant change in A1C level from before to after intervention, and the time-by-group interaction term was significant (P = 0.04), indicating significant group differences in change in A1C levels over time. A1C values of patients of physicians in group B declined, while the A1C values of patients of physicians in groups A (control) and C increased. An analogous model (data not shown) predicting postintervention A1C levels from each group and controlling for preintervention A1C, patient age, sex, and preintervention Charlson score showed similar results.

Table 2.

Change in A1C and LDL cholesterol testing rates and values

	All	Intervention group
		A	B	C
		Control	Learning cases	Cases plus expert
A1C testing rate (n)	2,020	691	725	604
12 months preintervention (%)	90.8	90.9	91.6	89.6
12 months postintervention (%)	89.2	90.2	88.9	88.2
Change in A1C test rate (%)	−1.6	−0.7	−2.7	−1.4
Time (P = 0.07)	Condition P = 0.79	Time-by-condition P = 0.63
Last A1C value (mean) (n)	1,686	578	601	507
12 months preintervention	7.38	7.33	7.47	7.32
12 months postintervention	7.45	7.39	7.46	7.50
Change in A1C value (%)	0.07	0.06	−0.01	0.18
Time P = 0.009	Condition P = 0.58	Time-by-condition P = 0.04
Last A1C value, patients on insulin (mean) (n)	526	172	199	155
12 month preintervention	7.95	7.87	8.02	7.95
12 month postintervention	7.92	7.88	7.85	8.03
Change in A1C value	−0.03	+0.01	−0.17	+0.08
Time P = 0.69	Condition P = 0.74	Time-by-condition P = 0.15
LDL cholesterol testing rate (n)	2,020	691	725	604
12 months preintervention (%)	71.6	74.2	70.2	70.2
12 months postintervention (%)	73.6	73.7	74.9	72.1
Change in LDL cholesterol test rate (%)	+2.0	−0.5	+4.7	+1.9
Time P = 0.15	Condition P = 0.76	Time-by-condition P = 0.30
Last LDL cholesterol value (mean) (n)*	1,178	412	419	347
12 month preintervention	106.0	107.0	106.3	104.5
12 month postintervention	102.6	102.9	103.9	100.8
Change in LDL cholesterol value	−3.4	−4.1	−2.4	−3.7
Time P < 0.001	Condition P = 0.63	Time-by-condition P = 0.67

*Those with test values both pre- and postintervention. †Those with A1C values and on insulin both pre- and postintervention.

Table 2 also shows results of the analysis restricted to patients on insulin for ≥120 days in both the pre- and postintervention periods. The overall time-by-condition term indicates no significant postintervention group differences in A1C change in insulin-treated patients. Patients not on insulin had an overall A1C increase of 0.13% from pre- to postintervention, with no differences by group (P = 0.10). Most of the A1C advantage of group B patients was concentrated in the subgroup using insulin.

Table 2 also shows that LDL cholesterol values improved over time (P < 0.001), with no significant differences in LDL cholesterol test rates (P = 0.30) or LDL cholesterol values (P = 0.67) across study groups. Statin use (P < 0.0001) and fibrate use (P = 0.001) increased over time with no differences across study groups (time-by-group P = 0.22 for statins and P = 0.78 for fibrates).

Among patients above A1C goal during the preintervention time period (n = 907), the rate of postintervention initiation or titration of blood glucose–lowering medications was 31.5% in group A, 32.6% in group B, and 36.8% in group C (P = 0.41). Insulin titration could not be ascertained from data sources. Among patients with their most recent preintervention LDL cholesterol ≥100 mg/dl (n = 701), initiation and titration of lipid medications was 20.8% in group A, 24.2% in group B, and 21.8% in group C (P = 0.66). Among patients with their most recent preintervention blood pressure >130/80 mmHg (n = 949), the rate of postintervention initiation or titration of blood pressure–lowering medications was 27.3% in group A, 24.7% in group B, and 28.2% in group C (P = 0.61). Thus, the intervention did not significantly increase rates of treatment intensification.

Table 3 provides data on risky prescribing events. While the time-by-condition term was only marginally significant (P = 0.08), a contrast of the control condition (group A) with the combined intervention groups (B plus C) showed a significant effect (P = 0.03) in favor of groups B and C for reduction in metformin use in patients with a renal contraindication to metformin use. No changes by group in metformin use following a CHF diagnosis (P = 0.36) were observed, consistent with the fact that this issue was not highlighted in the clinical cases.

Table 3.

Change in risky metformin-prescribing events

	All	Intervention group
		A	B	C
Metformin fill among those with a prior serum creatinine test ≥1.5 mg/dl (n)	264	77	100	87
12 months preintervention (%)	13.8	6.4	14.8	19.3
12 months postintervention (%)	6.7	6.4	4.8	9.0
Change in metformin fill rate (%)	−7.1	0	−10.0	−10.3
Time P = 0.002	Condition P = 0.23	Time-by-condition P = 0.08
Metformin fill following a CHF diagnosis (n)	259	73	107	79
12 months preintervention (%)	16.6	15.8	15.3	19.0
12 months postintervention (%)	12.6	16.4	11.6	10.5
Change in metformin fill rate (%)	−4.0	+0.6	−3.7	−8.5
Time P = 0.11	Condition P = 0.88	Time-by-condition P = 0.36

CONCLUSIONS

Despite the brevity of the learning interventions, we observed significant effects on diabetes care delivered to actual patients in the 12 months after each intervention was completed. First, both intervention groups showed an ∼60% decline in risky metformin use in patients with reduced renal function compared with the control group. Notably, there was no change in risky use of metformin in CHF patients, which was a clinical issue not included in the learning cases. Second, patients of physicians in group B had more desirable patterns of glycemic control postintervention than patients in the other groups. Finally, while lipid control improved in all groups, no differences in LDL cholesterol levels were observed across intervention groups.

Based on postintervention changes in treatment patterns in real patients, we conclude that different things were learned by the physicians in our two intervention groups (B and C). The part of each intervention that was common to both physician groups was comprised of three simulated cases, two of which focused on insulin initiation and titration. The data suggest, but do not prove, that physicians in group B learned to make additional insulin titration moves for their patients who were already on insulin (see Table 2), and better management of insulin-treated patients led to the A1C improvement we observed in group B.

The physician opinion leader did not enhance the simulated learning intervention. The reason for this is unclear but could have included the following: 1) the opinion leader focused on less-relevant clinical issues than the simulated program itself, 2) the presence of another physician observing increased anxiety of the physician learner, or 3) the positive reinforcement of the opinion leader led the providers to believe they did not need to change their clinical behaviors with actual patients. This finding is encouraging from a cost perspective because opinion leader time adds significant cost to the intervention.

There are a number of factors that limit the interpretation of these data. First, the study was conducted among PCPs at a single medical group, and generalization of results to other settings is not assured. Second, the lack of data on insulin intensification and patient medication adherence constrains our analysis of how interventions mediated their effects. Third, because blood pressure values were not yet electronically available at the time of data analysis, we were unable to assess the impact of the intervention on changes on blood pressure, although the learning cases included blood pressure management issues. Fourth, the study was conducted at a site with relatively good baseline diabetes care, and intervention impact may differ in settings where baseline care is poorer. Finally, the study design did not assess whether augmentation of the interventions using additional cases or periodic repetition could further amplify the benefits of this limited intervention involving only three diabetic case subjects.

Despite some limitations, study results are interesting and important. Safety improvement was substantial, and although glycemic control improvements were modest, few previous physician-learning interventions have reported positive results (25). This learning technology could be more effective if simulated cases were customized for each individual physician based on analysis of patterns of care in electronic health records. Further experimentation with this intervention strategy in other settings and for other clinical conditions is also warranted. Effective physician learning interventions such as these, which are brief, enjoyable, and scalable, may complement other care improvement strategies and may contribute to the essential goal of improving the safety and quality of chronic disease care.