Educating Resident Physicians Using Virtual Case-Based Simulation Improves Diabetes Management: A Randomized Controlled Trial

Abstract

Purpose

To test a virtual case-based Simulated Diabetes Education (SimDE) intervention developed to teach primary care residents how to manage diabetes.

Method

Nineteen primary care residency programs, with 341 volunteer residents in all post-graduate years (PGY), were randomly assigned to a SimDE intervention group or control group (CG). The web-based interactive educational intervention used computerized virtual patients who responded to provider actions through programmed simulation models. Eighteen distinct learning cases (L-cases) were assigned to SimDE residents over 6 months from 2010–2011. Impact was assessed using performance on 4 virtual assessment cases (A-cases), an objective knowledge test, and pre-post changes in self-assessed diabetes knowledge and confidence. Group comparisons were analyzed using generalized linear mixed models, controlling for clustering of residents within residency programs and differences in baseline knowledge.

Results

The percentage of residents appropriately achieving A-case composite clinical goals for glucose, blood pressure, and lipids was: A-Case 1, SimDE = 21.2%, CG = 1.8%, P = .002; A-Case 2, SimDE = 15.7%, CG = 4.7%, P = .02; A-Case 3, SimDE = 48.0%, CG = 10.4%, P < .001; A-Case 4, SimDE = 42.1%, CG = 18.7%, P = .004. The mean knowledge score and pre-post changes in self-assessed knowledge and confidence were significantly better for SimDE group than CG participants.

Conclusions

A virtual case-based simulated diabetes education intervention improved diabetes management skills, knowledge, and confidence for primary care residents.

The safety and quality of diabetes care in the United States is suboptimal, with less than 20% of diabetes patients simultaneously achieving recommended levels of glycated hemoglobin (A1c), blood pressure (BP), and low-density lipoprotein cholesterol (LDL).^1–3 In support of the need for provider education, several studies suggest that lack of timely intensification of treatment when patients are not at recommended clinical goals is a major obstacle to better diabetes care in the United States.^4–6 Some data suggest that problems related to timely and appropriate treatment are especially pronounced among resident physicians, with appropriate medication intensification occurring in only 21% of diabetes visits managed by residents.⁷

Inadequate longitudinal outpatient training experience with diabetes patients in residency clinics may contribute to suboptimal outpatient diabetes care quality among practicing physicians.^8,9 Restriction of resident work hours by the Accreditation Council for Graduate Medical Education (ACGME) may further decrease resident physicians’ outpatient chronic disease management experience.¹⁰ Accordingly, the Education Committee of the American College of Physicians has recommended substantial reforms to improve resident training in outpatient chronic disease care.¹¹ Simulation experiences are increasingly viewed as the most prominent innovation in medical education for improving patient care and safety.^12,13

Virtual simulation is a method of training widely used in non-medical industries that need high reliability and safety such as aviation, engineering, and the military.^14,15 Medical simulation training began out of similar safety and quality concerns in the 1960s with Resusci-Anne,¹⁶ and now numerous applications use partial or full human mannequins for training in emergency management, procedural skills, obstetrics, and surgery.^17–22 However, medical simulation could be adapted for teaching the cognitive tasks of chronic disease care management, either in postgraduate training or as a continuing medical educational activity for practicing providers to focus on the frequent changes in guidelines and therapy observed in recent years. A literature and internet search revealed no other interactive simulated diabetes educational activities as defined by the ability of the learner to be immersed in cognitive tasks such as pharmacologic prescribing over consecutive patient encounters as if it were a real-world experience.²³

Two previous randomized trials of prototypes of this virtual case-based simulated diabetes education in practicing primary care physicians demonstrated a modest but significant 0.2% improvement in actual patient A1c levels in intervention versus control patients (P = .04), and a 10% reduction in metformin use in patients with impaired renal function (P = .03).^24,25 The authors theorized that simulated diabetes education may be more potent for resident physicians who have less baseline knowledge and experience than practicing physicians. To study this theory, we developed and evaluated a comprehensive curriculum of 18 simulated interactive cases capable of teaching medical residents diabetes care management skills for a large variety of complex clinical presentations.

Method

Study objective

This cluster randomized trial tested the primary hypothesis that an online virtual case-based Simulated Diabetes Education (SimDE) intervention would increase the ability of primary care residents to appropriately and safely achieve evidence-based diabetes clinical goals, evaluated using a set of virtual patient assessment cases.

Study setting and participant population

We conducted residency program recruitment through invitations to residency programs nationally by email, phone, and listserv postings.²⁶ Nineteen residency programs agreed to participate and distributed study brochures to their residents to encourage voluntary study participation (our institutional review board prohibited mandatory resident participation). Subsequently, 341 of 723 eligible residents provided online informed consent and completed a baseline survey. Residency program faculty participated in an advisory board that guided us through operational aspects of recruitment and implementation. We kept the identity of participating residents confidential from residency program staff. Over the study implementation period from October 2010 through April 2011, three iPad raffles were conducted to incentivize residents to complete the learning and assessment cases. All consented residents who completed the study evaluation received a $50 gift card.

Randomization

To minimize the possibility of imbalance across study groups in key program factors, we stratified residency programs prior to randomization by specialty (family medicine or internal medicine), number of consented residents, and a composite score based on a residency program survey concerning annual frequency of diabetes specific educational conferences and presentations and rates of resident participation in endocrinology electives. Ten residency programs with 177 consented residents were subsequently assigned using a random number generator to SimDE and 9 programs with 164 consented residents to the control group (CG). See Figure 1 for study design and participant flow.

Figure 1.

Consort diagram outlining residency program participation, resident subject recruitment, randomization, participation, and analysis, from a randomized study of virtual, case-based simulation improves diabetes management, 2010–2011

^aResidency programs that participated in the study were Good Samaritan Hospital Internal Medicine (IM), Cincinnati, OH; Mankato Family Medicine (FM), Mankato, MN; University of Minnesota IM, Minneapolis, MN; Smiley’s FM, Minneapolis, MN; Monmouth Medical Center IM, Long Branch, NJ; Baystate Medical Center IM, Springfield, MA; Drexel University College of Medicine/Hahnemann University Hospital, FM, Philadelphia, PA; Wesley FM, Wichita, KS; Oklahoma University, Tulsa FM, Tulsa, OK; University of Missouri FM, Columbia, MO; North Memorial FM, Minneapolis, MN; Methodist Hospital FM, St. Louis Park, MN; Banner Good Samaritan/Phoenix VA IM, Phoenix, AZ; St. Joseph’s Hospital and Medical Center IM, Phoenix, AZ; Baptist Health System IM, Birmingham, AL; Sioux Falls FM, Sioux Falls, SD; United FM, St. Paul, MN; Fairview Hospital/Cleveland Clinic FM, Cleveland, OH; Conroe FM, Conroe, TX.

^bOut of 28 residents who responded to phone and e-mail inquiries about reasons for not completing learning and assessment cases, reasons included: lack of time due to resident duties (22), lack of time due to studying for board exams (2), lack of time due to personal/family commitments (1), lack of satisfaction with the learning program (1), being on an overseas clinical rotation (1), and battling with an illness (1).

Intervention description

SimDE residents had secure online access to a standardized set of 18 learning cases (L-cases) designed by the research team with input from local diabetes experts.²⁷ The curriculum was evidence-based, and consistent with the American Diabetes Association and Institute of Clinical Systems Improvement (ICSI) diabetes care guidelines.^28–30 Medications were referred to by generic names and were recommended only for uses approved by the U.S. Food and Drug Administration.

The intervention used an intuitive electronic health record (EHR)-like interface that allowed treatment of virtual patients over multiple encounters at any desired follow-up intervals within 180 days of simulated time.³¹ Training for all residents included a 5 minute training video on how to order labs, medications, diagnostic tests, lifestyle interventions, SMBG (self-monitored blood glucose) testing, referrals, and follow-up visits;³² and a sham case without diabetes educational content to practice and assess proficiency with the interface.

The 18 L-cases, which covered type 1 and type 2 diabetes, included diagnosis and pharmacologic and lifestyle management across a range of illness severity and co-morbid conditions. SimDE residents were explicitly challenged to safely achieve each virtual patient’s individualized A1c, BP, and LDL goals within 6 months of simulated time. Physiologic simulation modeling realistically predicted the clinical impact of all resident actions on SMBG, A1c, BP, and LDL over follow-up time intervals chosen by the learner. Between each L-case encounter, learners received personalized feedback automatically generated by the SimDE program using a rules engine to critique the learner’s actions and guide future actions.²⁴ The feedback between encounters contained links to more detailed diabetes care management information that could be viewed at the discretion of the learner. Details of the feedback and other aspects of the educational intervention have been previously described.^25,31,33 We asked SimDE residents to complete three L-cases each month, and cases could be started, stopped, and returned to, or repeated as many times as desired anytime during the intervention period.

Data sources, measurement, and outcome measures

At the time of consent, all subjects completed a baseline survey for demographics, postgraduate year (PGY) level, and self-assessment questions on different diabetes management skills and confidence in managing diabetes. Subsequently, residents received their random assignment and were sent a website link with logon information for all study activities. CG residents and SimDE residents received the same training on the computer interface. After the 6-month intervention period, we assigned all SimDE and CG subjects four virtual assessment cases (A-cases), a 10-question case-based knowledge test (scored as a sum of correct responses from 0–10), and the same self-assessment questions about knowledge and confidence asked at baseline. These A-cases used the same interface as that of the L-cases but included no learning feedback between encounters. All residents were challenged on A-cases to achieve all care goals safely within 6 months simulated time. For each A-case, binary variables were defined to indicate achievement of A1c, BP, and LDL goals and cases were also scored for whether goals were achieved appropriately and safely based on pre-determined criteria for each A-case. The multiple-choice knowledge test consisted of complex case scenarios related to blood sugar, BP, lipids, and safety. The knowledge questions were difficult and, in addition to factual knowledge, required choosing clinical priorities for patients with multiple issues. The test was developed by the study team and tested with practicing providers and certified diabetes nurse specialists to ensure clarity. SimDE residents also rated satisfaction with the L-cases, and reported the extent to which what they learned affected their actual practice.

Analytic methods

We used general linear mixed models with an identity link and generalized linear mixed models with a logit link to predict binary achievement goals of A1c, BP, LDL, and composite goals for each A-case; and the total score on the objective diabetes knowledge test and correct responses to the individual knowledge items. These models included a fixed effect for study arm, a composite baseline measure of self-assessed knowledge to control for differences at baseline, and a random intercept for residency program. The analysis of the post-intervention self-assessment questions about knowledge and confidence utilized a general linear mixed model containing a fixed effect for study arm, time (baseline or post-intervention), the study arm by time interaction, a random term for program, and a random term for provider to account for the repeated measurement of the self-assessment questions. We repeated analysis of the objective knowledge score and self-assessed diabetes knowledge questions after stratifying on PGY (1, 2, 3–4) to examine the patterns of effects within PGY. PGY3 and PGY 4 groups were combined because of expected low numbers of eligible PGY4 residents (usually a “chief resident” position). We used Bonferroni correction for multiple comparisons for primary outcomes of goal achievement and appropriate and safe goal achievement, but not for secondary outcomes (individual knowledge and self-assessment items or specific case criteria for inappropriate treatment). The study was powered to detect a difference of 40% vs. 25% in composite goal achievement at .8 (two tailed alpha = 0.05) based on enrollment of 360 subjects with follow up data on 240 subjects.

Results

Participating residency programs represented all seven regions of the American Medical Association Medical Student Section³⁴ and included 12 family medicine (FM) and 7 internal medicine (IM) programs. Four residency programs described themselves as community-based, 7 as affiliated with academic institutions, and 8 as both. The range in number of residents per program was 12–130 (median = 27), and the number of ambulatory residency clinics associated with the residency programs ranged from 1–13 (median = 1). The estimated number of patients with diabetes on a PGY3 resident’s patient panel ranged from 1–100 (median = 20). Study enrollment rates ranged from 22% to 94% of the total number of residents at participating residency programs (median 48%). Characteristics of consented residents are described in Table 1.

Table 1.

Baseline Characteristics of 341 Consented Primary Care Resident Physicians, from a Randomized Study of Virtual, Case-Based Simulation to Improve Diabetes Management, 2010–2011^a

Characteristics	All, no. (%)	Intervention, no. (%)	Control, no. (%)
Treatment group	341 (100)	177 (51.9)	164 (48.1)
Age, median (interquartile range)	29 (4)	30 (4)	29 (4)
Women	180/341 (52.8)	89/177 (50.3)	91/164 (55.5)
Race/ethnicityb
White	174/341 (51.0)	78/177 (44.1)	96/164 (58.5)
Asian	91/341 (26.7)	57/177 (32.2)	34/164 (20.7)
Black	16/341 (4.7)	9/177 (5.1)	7/164 (4.3)
Hispanic	24/341 (7.0)	9/177 (5.1)	15/164 (9.2)
Other	36/341 (10.6)	24/177 (13.6)	12/164 (7.3)
Specialty
Internal medicine	159/339 (46.9)	89/176 (50.6)	78/163 (47.9)
Family medicine	146/339 (43.1)	68/176 (38.6)	70/163 (42.9)
Medicine–pediatrics	20/339 (5.9)	10/176 (5.7)	10/163 (6.1)
Other	14/339 (4.1)	9/176 (5.1)	5/163 (3.1)
Post graduate year
1	125/341 (36.7)	69/177 (39.0)	56/164 (34.2)
2	105/341 (30.8)	53/177 (29.9)	53/164 (32.3)
3	101/341 (29.6)	52/177 (29.4)	48/164 (29.3)
4	10/341 (2.9)	3/177 (1.7)	7/164 (4.3)
Experienced an elective rotation with endocrinologist/diabetologist	62/335 (18.5)	35/174 (20.1)	27/161 (16.8)
Baseline knowledge self-assessment, moderate or very knowledgeablec
Knowledge using drugs to manage diabetesb	115/339 (33.9)	73/176 (41.5)	42/163 (25.8)
Knowledge adjusting insulin	120/339 (35.4)	70/176 (39.8)	50/163 (30.7)
Knowledge interpreting SMBGb	138/339 (40.7)	85/176 (48.3)	53/163 (32.5)
Knowledge setting individualized DM goals	116/339 (34.2)	68/176 (38.6)	48/163 (29.5)
Baseline confidence self-assessment, moderate or very confidentc
Confidence in managing individuals with DM	119/338 (35.2)	71/176 (40.3)	48/162 (29.6)

Abbreviations: SMBG = self-monitored blood glucose; DM = diabetes mellitus.

^aUnadjusted.

^bSignificant difference found at the .05 level for intervention vs. control, no adjustment for multiple comparisons. Among the 232 (97 intervention, 135 control) residents completing assessment cases, no significant differences by study arm were found.

^cScore on a 5-point scale (1 = not at all knowledgeable/confident, 2 = slightly, 3 = somewhat, 4 = moderately, 5 = very). Less than 3% of residents rated themselves a 5 (very knowledgeable or confident) in any category.

Reasonable proficiency with using the virtual interface was demonstrated by SimDE and CG residents prior to the start of the intervention and evaluation activities. Median time spent on the training and proficiency case was 8.5 minutes for SimDE residents and 9.5 minutes for CG residents (P = .32). Ten task activities were assigned to assess proficiency, and a score of 80% or higher on the 10 assigned tasks was demonstrated by 90/136 (67%) CG subjects and 72/97 (74%) intervention subjects.

L-cases were attempted by 80% of residents in the intervention group, and 50% completed all 18 L-cases. The median time spent per learning case was 16.8 minutes. The median time spent to complete all 18 cases was 5.2 hours, which was less than the median annual number of hours spent on diabetes specific conferences or presentations (8 hours annually) as reported in the residency program baseline survey.

232 residents (SimDE = 97 and CG = 135) completed at least one A-case. Table 2 describes each A-case’s hypothetical scenario and the results of the study for goal achievement for A1c, LDL, BP ignoring appropriate and safe criteria; goal achievement including appropriate and safe criteria; and the composite of these. The proportion of residents bringing each A-case to composite goal was significantly higher in the intervention group. Statistical significance was maintained using a Bonferroni-corrected P < .0125 (.05/4) for composite goal achievement and for composite appropriate and safe goal achievement. SimDE residents who partially completed 18 L-cases had improved A-case outcomes compared to the control, but not as good as SimDE residents who completed all cases (data not shown). Median time for completion of an A-case was 25.6 minutes and there was no consistent pattern observed for amount of time spent on assessment cases and ability to bring A-cases to care goals. Intervention effects were greater for glycemic control and lipid management, with less impact on BP control. Appropriate and safe treatment was observed significantly more often in the intervention group in the four A-cases for starting aspirin in patients with coronary heart disease, making nephrology referrals when warranted for renal disease, using clinical pharmacists in a geriatric polypharmacy situation, and keeping self-monitored blood glucoses >70 mg/dl. Controls did not perform better than intervention on any of the 41appropriate and safe criteria assessed. A subgroup analysis of A-case outcomes by PGY level (Supplemental Digital Table 1 [LWW INSERT LINK]) demonstrated that all post-graduate years had a benefit from the intervention, but the largest intervention effects were in PGY1.

Table 2.

Percentages of Primary Care Residents Meeting Outcome Goals and Criteria for Appropriate and Safe Treatment for Four Distinct Virtual Assessment Cases (A-Cases), from a Randomized Study of How Virtual, Case-Based Simulation Improves Diabetes Management, 2010–2011

Individualized case goal (and initial clinical info)	Outcomes assessed for goal achievement and meeting appropriate and safe criteria	% Intervention (n = 96)a	% Control (n = 135)a	P-value
Hypothetical A-Case #1: A 65 year old woman with T2DM, coronary heart disease, and kidney disease [GFR < 30 mg/dl] presenting for routine care.
A1c <8% (initial A1c 10.8% on no glycemia meds)	A1c goal ignoring appropriate/safe criteria	82.7	50.0	<.001
	Start basal insulin at appropriate dose	83.2	79.1	.46
	Avoid metformin	80.7	71.8	.15
	Avoid thiazolidinedione	84.6	92.0	.22
	A1c goal including above criteria	53.6	30.7	.003
LDL < 70 mg/dl (initial LDL 85 mg/dl on atorvastatin 40 mg/d and fenofibrate)	LDL goal ignoring appropriate/safe criteria	80.1	31.4	<.001
	Intensify statin therapy	85.2	36.5	<.001
	Discontinue fenofibrate	47.0	5.1	<.001
	Monitor liver enzymes	91.6	69.2	.003
	LDL goal including above criteria	39.4	3.1	<.001
BP < 140/90 mm Hg (initial BP 145/88 mm Hg on no BP meds)	BP goal ignoring appropriate/safe criteria	99.3	99.2	.92
	Make appropriate BP med choice	90.0	92.3	.64
	Monitor potassium and creatinine	92.7	91.7	.82
	Advised patient to stop NSAID	98.9	95.6	.20
	BP goal including above criteria	84.1	80.5	.53
Other clinical issues	Aspirin was started for CHD	76.1	51.3	.02
	Refer to nephrology	68.3	53.7	.04
	Refer for medical nutrition therapy	97.9	93.3	.16
Composite of above A1c, BP, LDL goals	Not including appropriate/safe criteria	74.7	20.3	<.001b
	Including appropriate/safe criteria	21.2	1.8	.002b
Hypothetical A-Case #2: A 37 year old male patient without known diabetes and on no medications presents with polydipsia, polyuria, and fatigue. Labs ordered will confirm T1DM diagnosis.
A1c <7% (initial A1c 12.7% on no meds)	A1c goal ignoring appropriate/safe criteria	95.7	77.9	.008
	Order urinalysis to rule out ketonuria	78.9	74.8	.56
	Avoid oral agents (type 1 diagnosis)	85.3	79.6	.3
	Start appropriate multidose insulin dose	63.8	43.8	.01
	Order glucagon kit	40.0	17.4	.002
	A1c goal including above criteria	23.3	7.1	.005
LDL < 100 mg/dl (initial LDL 108 mg/dl on no meds)	LDL goal ignoring appropriate/safe criteria	98.9	78.7	.009
	Initiate statin therapy	98.9	80.9	.01
	Order liver enzymes	86.0	69.9	.01
	LDL goal including above criteria	86.0	61.0	.006
BP < 140/90 mm Hg (initial confirmed BP 147/88 mm Hg)	BP goal ignoring appropriate/safe criteria	100	98.5	.51
	Start ACE or ARB	93.1	93.7	.88
	Order potassium and creatinine	92.4	94.9	.47
	BP goal including above criteria	87.2	89.2	.68
Other clinical issues	Order autoantibodies to confirm T1DM	53.0	49.3	.6
Composite of above A1c, BP, LDL goals	Not including appropriate/safe criteria	95.7	63.2	<.001b
	Including appropriate/safe criteria	15.7	4.7	.02
Hypothetical A-Case #3: A 71 year old male with T2DM and coronary heart disease on multiple medications presenting with fasting hypoglycemia, peripheral edema, and symptoms of obstructive sleep apnea.
A1c <8% (initial A1c 8.9% on pioglitazone and glargine)	A1c goal ignoring appropriate/safe criteria	98.9	94.0	.13
	Order sleep study	93.5	89.6	.35
	Discontinue pioglitazone	81.9	83.4	.78
	Avoid SMBG’s < 70 mg/dl	80.1	62.9	.02
	A1c goal including above criteria	62.4	44.6	.02
LDL < 70 mg/dl (initial LDL 77 mg/dl on simvastatin 40 mg/d)	LDL goal ignoring appropriate/safe criteria	85.0	38.2	<.001
	Statin therapy intensified	84.8	37.5	<.001
	Liver enzyme tests ordered	92.4	76.4	.03
	LDL goal including above criteria	80.2	28.8	<.001
BP < 140/90 mm Hg (initial BP 165/99 on HCTZ and metoprolol)	BP goal ignoring appropriate/safe criteria	97.8	81.6	.007
	Prescribe appropriate BP medication	82.2	67.8	.03
	Order potassium and creatinine	96.7	94.4	.44
	BP goal including above criteria	79.9	61.6	.01
Other clinical issues	Start aspirin	80.9	60.2	.006
	Refer to a clinical pharmacist	79.9	19.0	<.001
Composite of above A1c, BP, LDL goals	Not including appropriate/safe criteria	82.3	33.3	<.001b
	Including appropriate/safe criteria	48.0	10.4	<.001b
Hypothetical A-Case #4: A 54 year old woman with T2DM, heart disease, severe obesity [BMI 45], and tobacco use presenting with symptoms of depression and painful peripheral neuropathy.
A1c <8% (initial A1c 8.9%% on metformin, glargine, pioglitazone, and aspart)	A1c goal ignoring appropriate/safe criteria	100	95.3	.04
	Adjust insulin in safe increments	86.4	79.6	.22
	Discontinue pioglitazone (heart disease)	83.3	80.8	.69
	A1c goal including above criteria	72.8	58.2	.11
LDL < 100 mg/dl (initial LDL 88 mg/dl, not on a statin)	LDL goal ignoring appropriate/safe criteria	99.3	99.5	.85
	Initiate statin (even though at LDL goal)	80.8	47.0	<.001
	Order liver enzymes	90.9	70.9	.009
	LDL goal including above criteria	76.2	39.1	<.001
BP < 140/90 mm Hg (initial BP 165/99 mm Hg on no BP meds)	BP goal ignoring appropriate/safe criteria	97.7	87.6	.05
	Start 2 drugs (Stage 2 HTN diagnosis)	85.2	76.6	.14
	Start thiazide diuretic	88.2	76.0	.04
	Order potassium and creatinine	97.8	96.7	.65
	BP goal including above criteria	72.2	59.3	.07
Other clinical issues	Address neuropathy symptoms	73.8	66.3	.26
	Refer for bariatric surgery consultation	98.9	72.2	.003
	Address depression	95.4	94.0	.67
	Advise to stop smoking	70.1	77.5	.33
Composite of above A1c, BP, LDL goals	Not including appropriate/safe criteria	96.6	83.5	.02
	Including appropriate/safe criteria	42.1	18.7	.002b

Abbreviations: T2DM = type 2 diabetes; GFR = glomerular filtration rate; A1c = glycated hemoglobin; LDL = low density lipoprotein; BP = blood pressure; NSAID = nonsteroidal anti-inflammatory drug; CHD = coronary heart disease; T1DM = type 1 diabetes; SMBG = self-monitored blood glucose; ACE = angiotensin converting enzyme; ARB = angiotensin receptor blocker; HCTZ = hydrochlorothiazide; HTN = hypertension; BMI = body mass index.

^aNumerator raw numbers are not included because the percentages are model-based obtained from generalized mixed models that control for baseline knowledge and account for clustering of residents in the program.

^bP < .0125. Bonferroni correction for 4 cases to composite goal and 4 to composite goal appropriately.

Table 3 shows knowledge test results. Nine out of the 10 knowledge questions were answered correctly more often by SimDE residents compared to CG, with statistically significant results for 5/10 questions. Significant correlations were observed between knowledge test results and A-case composite goal achievement (all cases, P < .05).

Table 3.

Mean Scores and Performance Results on a 10 Question Multiple Choice Knowledge Test Completed by Primary Care Residents in the Intervention and Control Groups After the SimDE Intervention Period, from a Randomized Study of How Virtual, Case-Based Simulation Improves Diabetes Management, 2010–2011^a

Category	Intervention (n = 92)	Control (n = 128)	P-value
Total score (items correctly answered out of 10), mean (SD)	5.3 (1.8)	4.1 (1.6)	.005
5 or more correct responses, % of respondents achieving	45.7	15.6	<.001
Individual knowledge question results, % with correct answer
1. Screen for diabetes using acceptable criteria	74.9	75.5	.93
2. Start basal insulin; set individualized A1c goal	57.8	26.8	.001
3. Check ketones in newly diagnosed symptomatic patients and start insulin	24.8	29.3	.62
4. Reduce basal insulin due to nocturnal hypoglycemia	63.7	70.4	.39
5. Relax A1c target due to hypoglycemia unawareness in a patient with T1DM	56.1	32.7	.004
6. Start insulin in a newly diagnosed symptomatic patient with T2DM	34.7	11.4	.001
7. Use a loop diuretic rather than thiazide diuretic in a patient with renal insufficiency; prescribe statin monotherapy (rather than combination lipid therapy); discontinue metformin due to renal contraindication	42.8	18.6	.007
8. Initiate BP treatment (without confirmatory testing) in patient with BP>180/100 mm Hg; prescribe a statin in high CV risk patients regardless of LDL level	59.0	44.4	.05
9. Start a statin; screen for depression; start basal insulin	65.2	57.2	.28
10. Manage geriatric issues such as polypharmacy; screen for depression; address hypoglycemia; start a statin	47.3	41.7	.43

Abbreviations: SimDE = Simulated Diabetes Education; A1c = glycosylated hemoglobin; T1DM = type 1 diabetes; T2DM = type 2 diabetes; BP = blood pressure; CV = cardiovascular; LDL = low density lipoprotein.

^aPredicted means, proportions, and P-values from general and generalized linear mixed model predicting knowledge score or item from study arm and baseline self-assessed knowledge composite. No adjustment for multiple comparisons. Where percentages are reported, numerator raw numbers are not included because the percentages are model-based obtained from generalized mixed models that control for baseline knowledge and account for clustering of residents in the program.

Changes in self-assessed knowledge and confidence were significantly greater for SimDE than CG for all items assessed, for all PGY levels, with the majority (9/15) of these comparisons reaching statistical significance at a threshold alpha of .05 (Table 4).

Table 4.

Results by PGY Level Subgroups of Mean Objective Knowledge Test Scores (Post-Intervention) and Mean Self-Assessed Knowledge and Confidence in Managing Diabetes (Pre and Post-Intervention Periods), from a Randomized Study of How Virtual, Case-Based Simulation Improves Diabetes Management, 2010–2011

Outcome	PGYa	Intervention (n = 92)			Control (n = 128)			P-value
		Pre	Post	Changeb	Pre	Post	Changeb
Objective knowledge scorec	1	--	5.2	--	--	3.8	--	.008
	2	--	5.5	--	--	4.1	--	.009
	3–4	--	5.2	--	--	4.5	--	.14
	All PGY	--	5.3	--	--	4.1	--	.005
Knowledge about all available drugsc	1	2.7	3.5	.8	2.6	3	.4	.04
	2	3.4	3.6	.2	3	3	0	.57
	3–4	3.5	3.8	.3	3.4	3.3	−.1	.01
	All PGY	3.2	3.6	.4	3	3	0	.01
Knowledge about how to start and adjust insulinc	1	2.8	4	1.2	2.5	3	.5	.007
	2	3.4	4.1	.7	3.1	3.4	.3	.08
	3–4	3.5	4.3	.8	3.5	3.7	.2	.04
	All PGY	3.2	4.1	.9	3	3.3	.3	.002
Knowledge interpreting patient SMBGsd	1	3.1	4.3	1.2	2.7	3.4	.07	.04
	2	3.4	4.2	.8	3.1	3.5	.4	.06
	3–4	3.7	4.2	.5	3.5	3.8	.3	.46
	All PGY	3.4	4.2	.8	3.1	3.6	.5	.02
Knowledge about setting individualized treatment goalsd	1	3	4	1	2.5	3.1	.6	.12
	2	3.2	4	.8	3.1	3.4	.3	.04
	3–4	3.5	4.2	.8	3.6	3.8	.3	.06
	All PGY	3.2	4.1	.9	3	3.4	.4	.008
Confidence in managing diabetesd	1	2.7	3.8	1.1	2.5	3.1	.6	.05
	2	3.4	4.1	.7	3	3.2	.2	.02
	3–4	3.6	4.1	.5	3.5	3.6	.1	.06
	All PGY	3.2	4	.8	3	3.3	.3	.005

Abbreviations: PGY = postgraduate year, SMBG = self-monitored blood glucose.

^aSample sizes for PGY subgroups: PGY1 n = 76; PGY2 n = 76, PGY3–4 n = 69.

^bFor change analysis, predicted means and p-values from general linear mixed models predicting item from study arm, time, and their interaction. P-value for interaction term. No adjustment for multiple comparisons.

^cMean number of questions correct out of 10.

^dMean rating on a 5 point Likert scale. Higher score indicates higher self-rated knowledge or confidence.

Results of the mixed quantitative and qualitative analysis of satisfaction in the SimDE group have been previously published.³¹ Likert responses on satisfaction items were favorably higher than neutral for all areas assessed including general satisfaction (93%) and willingness to recommend it to colleagues (91%). In addition, 77% of residents said they applied the learning to actual patients, 63% said they shortened visit intervals, 78% indicated they were more likely to add or increase drugs if their patients were not at goal, and 92% indicated they were more confident about insulin use in actual patients. Residents commented positively on the help the intervention provided with learning general aspects of diabetes management, insulin management, and achieving individualized goals.

Discussion

This study demonstrated that virtual case-based SimDE significantly improved the ability of resident physicians to achieve patient care goals appropriately and safely in virtual situations. It also improved objective measures of diabetes care knowledge and resulted in greater self-confidence in important aspects of diabetes management, including use of insulin, interpreting blood sugars, and individualizing treatment goals.³¹ SimDE residents said they were more likely to add medications and shorten visit intervals when seeing actual patients not at clinical goals.³¹ The learning intervention required about 1 hour of resident time each month for 6 months, was delivered as an adjunct to existing residency activities, and was well-liked by residents.

Previous randomized trials of similar simulated diabetes education for practicing primary care physicians demonstrated improved quality of care and safety outcomes in real patients.^24,25 Actual patient data was not permitted in this study in order to protect resident confidentiality at residency sties and because of low continuity of care in most ambulatory residency settings, but innovative methods were deployed to evaluate provider performance using virtual A-cases. The validity of this virtual assessment methodology is supported by significant positive correlations between mean objective knowledge score and composite outcomes for all A-cases.

Simulation modeling enables systematic and detailed evaluation of key aspects of clinical practice and safety issues using the A-cases. Examples of clinical practices that significantly improved on virtual assessment included statin use in patients not at LDL goal, aspirin use in patients with CHD, nephrology referrals in patients with renal disease, starting multi-dose insulin at appropriate doses, avoiding SMBGs < 70 mg/dl, and providing glucagon kits when indicated. Performance gaps identified through this type of virtual assessment suggest areas in need of improvement and could be used to direct future learning activities as envisioned in the practice-based learning interventions recommended by the ACGME.³⁵

Of note is that even though the intervention significantly improved most outcomes, post-intervention clinical performance in both groups was far from ideal, raising questions about the adequacy of current residency training methods and the clinical competency of graduates. The low scoring on assessment cases for achieving goals safely and appropriately was likely due in part to the difficulty of achieving the measured composite criteria, requiring 15–18 evidence-based guideline driven criteria to be met per case. Nevertheless, the differences between measures of goal achievement with and without the appropriate and safe criteria were dramatic and suggest that inappropriate and unsafe care may occur frequently in practice. Our data raise concerns that “goal-driven” accountability measures, which do not consider appropriateness of treatment, might lead to higher rates of risky prescribing events.

Previous studies show that many graduates of primary care residency programs are inadequately trained in outpatient diabetes care and often delay appropriate and timely intensification of treatment such as insulin.⁷ Most residency programs still emphasize inpatient training; and limited longitudinal continuity-of-care experiences, limits on residency work hours, and reduced funding to support faculty teaching time are other barriers to adequate training in outpatient chronic disease care.³⁶ Educational activity using simulation models can help address these barriers because of the economy, brevity, standardization of learning content, and personalization of the experience. Moreover, simulation models can quickly incorporate new evidence-based recommendations as knowledge evolves. Finally, this type of learning intervention can be implemented conjointly with a wide range of other strategies designed to improve quality of chronic disease care,^37,38 can be delivered conveniently via internet, and is scalable to unlimited numbers of residency sites at low marginal costs relative to the work involved in updating and enhancing the educational experience as guidelines change. The virtual cases can be used as an adjunct to other residency training activity, and our findings suggest that the education may work best for those with lower baseline knowledge and experience, suggesting possible benefits of extending this learning strategy to medical students or other health science students.

Although the virtual encounters cannot authentically replicate all of the potential challenges of face-to-face meetings with real patients, many common challenges to patient self-management such as pharmacologic and behavioral adherence, depression, and variation in behavioral readiness to change were incorporated into the simulation model. Realism was magnified by allowing each learner to follow a unique trajectory based on actions taken, including the potential to cause harm and for remedial actions based on corrective feedback received between encounters. SimDE residents who partially completed 18 L-cases had improved A-case outcomes compared to the control, but not as good as SimDE residents who completed all cases. More research is needed to determine if a smaller number of more focused L-cases could result in similar outcomes to those observed by this study, or if a larger number of cases could result in better outcomes. The benefit of simulation in assuring that trainees are exposed to a wide variety of virtual patients in addition to their actual patients is an intriguing area for medical education research.

Our analysis is limited by selective participation in the study and incomplete L-case completion. Further evaluation would be warranted to assess efficacy if such education were to be mandatory in residency training. Physicians are generally considered poor at self-assessment, and another limitation of our study is the use of self-assessment to evaluate changes in knowledge and self-confidence in managing diabetes.^39,40 While the SimDE group had greater increases in self-assessed confidence and perceived knowledge, the more reliable outcomes observed for objective knowledge scores and performance on assessment cases is reassuring.

Bias could also be introduced because A-case completion was lower for SimDE participants (55%) than CG (82%). A-case non-completers identified lack of time as the principal barrier to case completion, and total time commitment in this study was about 8 hours for SimDE and about 2 hours for CG, making it more likely for the CG group to find time for the assessment cases. A sensitivity analysis of A-case completers and non-completers revealed no statistically significant differences on baseline characteristics, including baseline knowledge, between the two groups, further supporting lack of time rather than other resident characteristics as the main barrier to A-case completion.

This study was limited to resident physicians, but other trainees such as nurse practitioners, physician assistants, pharmacists, and practicing physicians may also benefit from virtual case-based simulated education.⁴¹ Such education is customized to each learner’s actions within each case, resulting in unique learning trajectories, and such highly personalized and interactive education (e.g., learning by doing) is more powerful and efficient than didactic education based on current adult learning theories.^42–44

Virtual case-based SimDE enhanced measures of resident performance, improved objective and subjective knowledge scores, and raised self-confidence in managing patients with diabetes. Broader use of such virtual diabetes education using simulation models may be warranted to improve the much needed diabetes care capabilities of the future healthcare workforce. In addition, the innovative virtual patient outcome assessment method may prove useful in other settings where actual patient outcomes are difficult or costly to measure.