Update in Medical Education 2007

INTRODUCTION

In this article, we summarize articles selected for presentation at the Update in Medical Education at the 31st annual meeting of the Society of General Internal Medicine.

METHODS

Our search was limited to articles published between January 1, 2007 and December 31, 2007 in one of 14 publications: Academic Medicine, American Journal of Medicine, Annals of Internal Medicine, BMJ, JAMA, Journal of the American Geriatrics Society, Journal of Continuing Education in the Health Professions, Journal of General Internal Medicine, Lancet, Medical Education, Medical Teacher, New England Journal of Medicine, Teaching and Learning in Medicine and Quality and Safety in Health Care. We defined “medical education” broadly and included all study designs and levels of learners. Final articles selected for presentation were chosen by the authors based on consensus after a discussion centered on the questions “Is this relevant to academic general internists?” and “Will this article change the practice of teaching and learning?”

RESULTS

Twenty two articles were selected for presentation at the SGIM meeting. We present ten in this article; short summaries of the other twelve appear in the online Appendix. Presented reports fall into four themes: (1) educational interventions, (2) continuity, (3) clinical reasoning and (4) professionalism and are reviewed below.

Educational Interventions

Leenstra JL, Beckman TJ, Reed DA, Mundell WC, Thomas KG, Krajicek BJ, Cha SS, Kolars JC, McDonald FS. Validation of a method for assessing resident physicians’ quality improvement proposals. JGIM. 2007; 22:1330–1334.

Residencies are involving trainees in quality improvement (QI) projects as a way to address the Accreditation Council on Graduate Medical Education (ACGME) competencies, yet there are no published methods for assessing resident QI proposals. The aim of this study was to create and assess the validity of a tool for evaluating resident QI proposals.

The content of the QI proposal assessment tool was decided by experts based on consensus findings of the 2004 Annual Achieving Competence Today Conference. Between July and December 2005, the initial instrument was tested by faculty and residents on a monthly basis as part of the quality improvement curriculum for Mayo Clinic internal medicine residents. The instrument was modified through this iterative process, and the final 7-item Quality Improvement Proposal Assessment Tool (QIPAT-7) was developed. Items were structured on a 5-point rating scale. Following IRB approval, the instrument was pilot tested by five faculty and two chief medical residents on three randomly chosen resident QI projects from the prior year. Raters met and resolved all differences in assigned ratings. It was decided that in order to receive a score of ≥ 3 for an item, all anchor descriptors should be achieved. The QIPAT-7 was then used to score 45 consecutive resident QI projects between July 2004 and July 2005.

Principal factor analysis demonstrated that each of the seven items represented one dimension. Item mean scores ranged from 1.9 to 3.4 on the 5-point scale. Excellent interrater reliability for each item (range 0.79–0.93) and internal consistency reliability among the items (Cronbach’s alpha = 0.87) was demonstrated.

Limitations included that this instrument was developed at a single institution where residents received an intensive 64-hour QI curriculum. In addition, the raters used to establish instrument reliability were familiar with QI concepts and were integral to scale development.

In conclusion, the QIPAT-7 is supported by content and internal structure validity evidence, and is a useful and exportable tool for evaluation of QI proposals. Further studies are needed to determine whether proposal scores correlate with successful project implementation and outcomes.

O’Mahoney SO, Mazur E, Charney P, Wang Y, Fine J. Use of multidisciplinary rounds to simultaneously improve quality outcomes, enhance resident education and shorten length of stay. JGIM 2007; 22:1073–1079.

ACGME core competencies recognize the importance of training in systems based care and the delivery of evidence-based care. This study sought to determine whether resident-centered multidisciplinary rounds (MDR) lead to measurable improvement on core measures embraced by JCAHO and CMS while improving length of stay and enhancing resident education.

This study was conducted on 44 residents at a university affiliated community teaching hospital. The multidisciplinary group involved in rounds included members of the internal medicine service, case managers, nurse coordinators, dieticians, pharmacists and representatives from physical medicine and psychiatric services. MDR was led by the chief of medicine and a clinician educator and was conducted for 1 hour three times a week. Resident teams were present only for discussion of their own patients and discussions were focused on relevant issues. Outcomes on core measures for specific diagnoses (congestive heart failure, acute myocardial infarction and community acquired pneumonia) were chosen as the focus for MDR and assessment. Length of stay data was sorted by diagnosis and medical attending from administrative databases. Data from July 2002 to June 2003 prior to MDR were compared to post implementation data from July 2003 to June 2004. Residents completed anonymous surveys regarding self-reported knowledge and attitudes about MDR after completing two blocks of MDR.

Analysis of performance on core measures revealed improvement in all three targeted diagnoses and in 5/10 of the individual core measures across all three diagnoses after implementation of MDR. Length of stay decreased 0.5 (95% CI 0.1–0.8) days for patients with targeted diagnoses and by 0.6 (0.5–0.7) days for all medicine DRGs. All medicine residents completed the survey and reported increased knowledge regarding core measures, system- based care and communication after the implementation of MDR. They also agreed that MDR improved efficiency, delivery of evidence-based care and relationships with involved disciplines.

Limitations include that this study was conducted at a single institution around three specific core diagnoses and relevant core measures. The measure of resident knowledge was based on self-perception of knowledge and both pre-MDR attitudes and knowledge measures were based on recall.

In conclusion, this MDR model of three-times-a-week, hour-long rounds presents an attractive model for achieving goals of resident education regarding systems based care, quality improvement on core measures and decreasing length of stay. Other institutions should consider replication of this model to determine if similar outcomes can be achieved.

Bechtold ML, Scott S, Nelson K, Cox KR, Dellsperger KC, Hall LW. Educational quality improvement report: outcomes from a revised morbidity and mortality format that emphasized patient safety. Qual Saf Health Care 2007; 16:422–427.

Traditional morbidity and mortality conference (MMC) tends to focus on individual actions and may interfere with identification of systems issues. Instead, MMC could be used to develop ACGME competencies of practice based learning and improvement and systems based practice. This study sought to describe and evaluate the implementation and outcomes of a redesign from a traditional MMC to a new monthly, patient safety morbidity and mortality conference (PSMMC)

The redesign process was conducted by an eight person multidisciplinary team. The following were the key goals for the PSMMC: 1) To teach system thinking to residents, fellows and faculty; 2) To provide a forum for discussion of adverse events and reasons contributing to occurrence; 3) To assist in the transformation of departmental culture to one which values patient safety and QI; and lastly 4) To expand knowledge and skills through a modified root cause analysis process. Outcomes assessed included system improvements generated from the PSMMC; attendance at PSMMC, and attitudes of residents and fellows regarding patient safety as measured on a pre-post 20-item 5-point Likert scale survey administered prior to the new conference and following the 8th month of the new conference.

Through the cases discussed in the 11 months, participants made 121 system improvement recommendations. Based on the likelihood of achieving high impact changes, facilitators determined 39 (32%) of the recommendations should be pursued. Of these targeted changes, 23 (59%) were fully implemented; 11 (28%) were partially implemented and 5 (13%) were abandoned due to impracticality or redundancy. Average attendance increased from 41 ± 8 to 50 ± 10 participants ( < 0.03). Of 111 residents and fellows, 58 completed the pre-post attitudinal survey. Six of 20 items showed substantial change with four occurring in the desired direction, and 11/14 remaining items trending in the desired direction.

Limitations include that this was conducted at a single site and that the actual impact on patient safety was not measured.

In conclusion, this study demonstrated that an educational intervention could bring about increased participation of residents and some modest change in attitudes while enhancing system performance in hopes of leading to improved patient outcomes.

Haist SA, Wilson JF, Lineberry MJ, Griffith CH. A randomized controlled trial using insinuated standardized patients to assess residents’ domestic violence skills following a two hour-workshop. Teach Learn Med 2007; 19(4): 336–342

Prior research on impact of resident education regarding domestic violence (DV) has neither randomized individual residents to an intervention nor examined the impact on clinical care. The aim of this randomized controlled trial was to assess the impact of a two-hour DV workshop on clinical practice of residents through use of insinuated standardized patients (paid, standardized actors appearing in clinical practice as actual patients).

Twenty-seven volunteer internal medicine residents from a single institution were randomized to a two-hour interactive DV workshop or a control (chronic pain) workshop. Outcomes were measured by thirteen standardized patients (SPs) that were recruited and were trained on two DV cases. Reliability of their evaluations was established. SPs portrayed either a 27- year-old female with a shoulder injury or a 27- year-old female with six weeks of tiredness and sadness. These SPs were then insinuated into resident continuity clinic 1–3 months post workshop and/or 4–7 months post workshop. Clinical care given by the residents was assessed using a checklist completed immediately after the clinic visit by the SP. Residents were assessed on whether or not DV was identified, performance on 12 (injury case) or 14 (depression case) checklist items, and performance on eight DV safety plan counseling items.

The DV workshop residents did not identify DV as an issue in SPs any more frequently than did control residents (64% vs. 56%,  = 0.86) However, the DV trained residents performed better (greater than 75% of items) on the DV checklist items compared to control residents (36% vs. 9%,  = 0.04) and on safety plan counseling (greater than 75% of items) (40% vs. 13%, p = 0.04).

Limitations include that this was a single site study with a voluntary and paid group of residents. Also residents were aware that SPs would be used to assess workshop skills in their clinic settings.

In conclusion, after a DV patient was identified, DV workshop trained residents provided better clinical care to insinuated SPs than did control residents. Other interventions will need to be assessed to enhance identification of DV victims and to further improve clinical care delivered to these women. Insinuated SPs can be successfully used to assess clinical care outcomes of educational interventions.

Chossis I, Lane C, Gache P, Michaud PA, Pécoud A, Rollnick S, Daeppen JB. Effect of training on primary care residents’ performance in brief alcohol intervention: a randomized controlled trial. JGIM. 2007 Aug; 22(8):1144–9.

Prior work has demonstrated a reduction of alcohol consumption in non-dependent drinkers using the brief alcohol intervention (BAI) model. Previous studies have shown BAI training can change physician behavior, but these studies were not blinded or controlled and did not assess patient outcomes. These authors performed a randomized trial of resident education regarding BAI hypothesizing that trained residents would increase BAI component use and patients of trained residents would reduce hazardous drinking.

Primary care residents were assigned to the interventional BAI training or to a traditional didactic program on lipid management. Five residents with prior training in alcohol treatment were excluded. BAI training occurred in two sessions delivered in two half days that were two days apart. In the first session, residents were given didactic information on the BAI model followed by videotape demonstration and role-play. Residents were then given a summary checklist of BAI components, a textbook on alcohol, and patient educational materials. In the second session, residents practiced with trained standardized patients.

Beginning one week after training, consecutive patients received a self administered questionnaire on alcohol, tobacco, drug use, cholesterol, immunizations, depression and accidents. Patients were selected for study inclusion if ≥19 years old, had a scheduled appointment, and were hazardous drinkers in the past 12 months. Patients were blinded to the aims of the study. Hazardous drinking was defined as >14 drinks/week and/or >4 drinks/occasion for men <65 and >7 drinks/week and/or >3 drinks/occasion for women and for men >65. Patients received a feedback form summarizing results of their questionnaire and were instructed to give the form to the resident during the visit. Hazardous drinkers were interviewed after the visit to determine the type and number of BAI components residents conducted, had an AUDIT assessment, and interviewed by telephone three months after the visit to assess alcohol intake.

There were 13 residents in the intervention and control groups. Two-hundred and sixty patients were enrolled and interviewed after the visit. There was no statistical difference in whether or not residents addressed alcohol consumption (BAI 54% vs. control residents 46%). BAI residents performed more BAI components than controls (2.4 vs. 1.5 p = 0.001). BAI residents were more likely to explain safe drinking limits (27% vs. 10% p = 0.001), provide feedback to patients on alcohol use (33% vs. 21% p = 0.03), and to ask patient opinions on safe drinking limits (19% vs. 6% p = 0.02). There were no differences between residents on the other BAI components, which were performed between 4% and 22% of encounters. Two hundred and nineteen patients completed 3-month follow-up. Patients with interval visits had no difference in BAI components between intervention and control residents. In both groups, 37% of hazardous drinkers had become low risk drinkers with no significant difference between groups.

One limitation of the study may be that the patient questionnaire and feedback instrument and post visit interviews that occurred for all study patients may have been more powerful in affecting the prevalence of alcohol counseling and the reduction in hazardous drinking than the specific BAI behaviors presented and measured. This may overshadow any potential impact of the resident counseling on patient behaviors.

In conclusion, the educational intervention did not impact whether or not residents addressed alcohol consumption with patients. However, when consumption was addressed, trained residents discussed more BAI components with patients than controls. This did not result in differences in patient outcomes. Patient questionnaires and their analysis may affect patient outcomes more than detailed educational interventions with residents.

Continuity

Sisson SD, Boonyasai R, Baker-Genaw K, Silverstein J. Continuity clinic satisfaction and valuation in residency training. JGIM 2007 Dec; 22(12):1704–10.

Declining interest in primary care has focused attention on the quality of continuity clinic experiences and the potential negative impact of current clinics on career decisions. Recent recommendations from the Association of Program Directors in Internal Medicine (APDIM) and the American College of Physicians (ACP) have raised concerns about resident continuity clinic, referring to practices as “dysfunctional” and “inadequate”¹. These authors surveyed residents in three medicine training programs on their perspectives on continuity clinic experiences to better define the problem.

Residents were surveyed electronically through the Johns Hopkins Internet Learning Center. Each training program is a purchaser of the Hopkins online curriculum. Survey questions focused on the learning environment, patient mix and teaching quality in continuity resident practice. Of 260 residents, 218 (83.8%) completed the survey. All three study sites utilized electronic medical records (EMRs).

Residents rated their clinics as least valuable as compared to training in the wards and intensive care units. Preceptor characteristics had the greatest impact on resident value of clinic. Clinic operations, patient volume and characteristics, and resident career plans had a much smaller impact. In multivariate analysis, the greatest associations with resident value of clinic were with preceptor characteristics. Specific predictive preceptor characteristics were preceptors rated as good role models, and preceptor awareness of clinic and social resources. The only clinic operational characteristic that predicted resident value of clinic was whether patient flow was smooth. The only patient characteristic associated with resident value of clinic was a wide range of ages of patients.

Limitations include the fact that none of these residencies have primary care programs and that as few as 10–25% of residents in one program intended to pursue a generalist career. In addition, the survey did not distinguish between primary care and hospitalist generalist career intentions and therefore was unable to measure whether resident intent to practice outpatient primary care was associated with resident value of clinic. The survey measured only resident perceptions of clinic and did not objectively measure variables of clinic throughput, preceptor teaching skills, or patient characteristics. The survey could not assess the importance of an EMR in the resident’s experience since all sites used them. Finally, this survey was not designed to assess resident ambulatory knowledge or skills.

Maximal resident satisfaction may depend more on preceptor characteristics than clinic or patient characteristics. Programs may therefore wish to focus on preceptor selection and faculty development as key elements in improving resident clinic satisfaction.

Greene J, Rogers VW, Yedidia MJ. The impact of implementing a chronic care residency training initiative on asthma outcomes. Acad Med. 2007 Feb; 82(2):161–7.

This study sought to assess the impact of chronic care model (CCM) training upon resident use of the asthma chronic care model and on their continuity clinic patients’ resource use.

The Maine Medical Center CCM was implemented in resident continuity clinics between July 2002 and December 2003. The 59 participating residents included 13 in pediatrics, 27 in internal medicine, two in medicine/pediatrics, and 17 in family medicine. Residents completed a pre-intervention and post-intervention survey on facility of access to components of the CCM and on their patients’ access to calls to their home on asthma management, an asthma management plan, and enrollment in a self-management course or support group. Control residents (those not explicitly trained in CCM) were drawn from eight other CCM institutions participating in the national chronic care initiative, where CCM was not integrated into the residency of the control resident’s specialty. Patient outcomes were assessed by measuring visits to the emergency department.

Residents in the intervention group were more likely to access four of the twelve chronic care model elements, including access to detailed information on community programs, all or most patients receiving individual asthma plans, ease of access to asthma guidelines, and timely pharmacist input on complex regimens. Patients cared for by intervention residents had substantial decreased ER utilization compared with other asthma patients seen at the Maine Medical Center. For both pediatric and adult patients of enrolled residents, there was a decrease in ER visits (42.3% and 43.8%) as compared to increases in asthma ER visits for other patients (pediatric: 8.4%, adult: 2.9%).

Limitations include that comparisons in patient ER visits at different sites may be impacted by other variables not measured. Further, no information on sustainability was provided and one cannot assess the impact and importance of specific components of the chronic disease model.

In conclusion, this study suggests that a chronic disease training program may improve health outcomes and reduce costs of care for residents’ patients.

Ogur B, Hirsh D, Krupat E, Bor D. The Harvard Medical School-Cambridge integrated clerkship: an innovative model of clinical education. Acad Med. 2007 Apr; 82(4):397–404.

Current medical school clerkship models provide fragmented clinical experiences with little continuity of patient care or mentoring relationships. In a related paper, an overlapping group of authors asserted benefits of enhanced continuity in student education upon outcomes of skill development (i.e., chronic disease management), professionalism, patient-centered care, core doctoring skills, enhanced lifelong learning, and inter-professional understanding and collaboration². This paper presented preliminary results of an integrated longitudinal model across core clinical clerkships.

Of 189 rising 3rd year medical students at Harvard University, eight of 18 volunteers enrolled in a year-long program in which students were paired with faculty members in internal medicine, neurology, obstetrics and gynecology, pediatrics, and psychiatry. Students worked with these preceptors in longitudinal ambulatory care for the year. Each student followed a panel of patients throughout the continuum of care and also spent six weeks working with a surgeon. To evaluate the impact of this program, intervention students were compared to control volunteer students and to the body of remaining students in their class.

The clerkship achieved the goal of continuity. Of intervention students, 100% reported that they had seen patients very often or often before admission as compared to only 20% of comparison students (p < 0.001). Of intervention students, 100% had seen patients after discharge compared with 10% of control students (p < 0.001). Intervention patient cohort sizes ranged from 46–115 patients per student. Students followed at least ten patients through pregnancy and delivery. Intervention students received the majority of their feedback from faculty (88.1% vs. 31.5%) and the majority of their mentoring from faculty (77.55% vs. 37%). Intervention students self assessed themselves as better prepared than did control students to be caring with patients, to deal with ethical dilemmas, to understand social context of care, to involve patients in decision making, to relate well to a diverse population, and to be self-reflective, though there was no significant differences in NBME shelf exams or summative multi-station OSCEs.

In conclusion, this intriguing model provides a template for marked innovation in student teaching, but the generalizability of these findings is unclear. Program implementation occurred in a small community hospital; it is not clear that if this could be replicated in larger programs where coordination and cost might be prohibitive. Finally, participating students volunteered for the program; the experience might have been different with unselected students.

Clinical Reasoning

Peltier D, Regan-Smith M, Wofford J, Whelton S, Kennebecks G, Carney PA. Teaching focused histories and physical exams in ambulatory care: A multi-institutional randomized trial. Teach Learn Med. 2007; 19:244–250

While third-year medical students are generally prepared to perform complete history and physical examinations when beginning clinical clerkships, they may not yet have any experience focusing these to the patient’s concern. Precepting such students in the time pressured ambulatory care setting can be challenging for faculty. This study sought to measure the impact of “focus scripts” on student documentation of history and physical examination findings in outpatient progress notes. Scripts are cognitive structures or frameworks that can help organize elaborate knowledge bases into simpler models or patterns³, and have been shown to improve student teaching⁴.

Faculty and students at Dartmouth Medical School developed and pilot tested generic ‘acute’ and ‘chronic’ focus scripts as well as scripts specific to common acute (shoulder pain, knee pain, headache, dizziness, chest pain, dyspnea, and palpitations) and chronic (congestive heart failure, atrial fibrillation, hypercholesterolemia and noninsulin dependent diabetes mellitus) ambulatory care problems. In 2004–2005, third-year medical students on their ambulatory internal medicine clerkship at two different urban medical schools were randomized to receive ( = 29) or not receive ( = 31) the focus scripts intervention. Students submitted one write up each from the first week of their clerkship for blinded review and scoring by a faculty member at Dartmouth. Sixty progress notes were reviewed for 11 variables selected from existing, published criteria for utility^5,6 using a two-point scale from 0 (variable not present) to 1 (variable present).

Variables that improved in the intervention group included appropriate history (0.76 vs. 0.52,  = 0.05), physical exam (0.9 vs. 0.58,  = 0.005), analysis of lab values (0.24 vs. 0.03,  = 0.02), incorporation of lab values into diagnostic plan (0.35 vs. 0.13,  = 0.05), clear diagnosis (0.31 vs. 0.03,  = 0.003), and overall note total score (0.57 vs. 0.38, p < 0.001). Variables not significantly different included problem clarity, patient stability, appropriateness of differential diagnosis, clarity of patient plan, appropriateness of treatment given severity of illness, and cross-cover utility of the note. Documented improvement in progress note elements included in the scripts (history and physical exam features) as well as in elements not included (such as analysis of lab values and clear diagnoses) suggests that ‘focus scripts’ may facilitate the development of patterns of clinical information, an important part of clinical reasoning ability.

Study limitations included data analysis of only 60 notes from two schools and that schools which use template note formats will not lend themselves to use of this instructional method. In addition, the authors could not control for preceptor style of instruction and thus can only assume the variable influence to be random and equally distributed.

In conclusion, this study provides evidence that learning tools such as focus scripts may facilitate the specific task of documenting focused, appropriate evaluations in acute and chronic office visits by medical students.

Professionalism

Lurie SJ, Lambert DR, Nofziger AC, Epstein RM, Grady-Weliky TA. Relationship between peer assessment during medical school, dean’s letter rankings, and ratings by internship directors. JGIM. 2007; 22:13–16.

The dean’s letter or Medical Student Performance Evaluation (MSPE) provides a summary of students’ performance during medical school and is, therefore, more a letter of evaluation rather than of recommendation. As recommended by the Association of American Medical Colleges, MSPEs often include some ranking of students in comparison to their peers; such MSPE rankings of graduates are closely related to residency program director’s (PD) later evaluations⁷. The authors sought to compare peer assessment ratings of work habits (WH) and interpersonal attributes (IA) of second- and third-year medical students against later MSPE rankings and ratings by PDs.

Two hundred eighty one medical students who graduated in 2004, 2005 or 2006 from the University of Rochester School of Medicine and Dentistry and who had participated in peer assessment exercises during second and third year were included. In the MSPE, students were ranked into one of four categories based upon weighted grades in required clinical clerkships. Approximately 20% were ranked outstanding, 25% excellent, 50%–55% very good and < 5% good. Standardized peer assessment forms which included independent items assessing WH and IA were completed anonymously by students. Each student rated a different group of six to twelve classmates once during both the second and third years of medical school. A 15-item survey was sent to PDs approximately 10 months after the class of 2004 had graduated and included items to rate graduates on general clinical, interpersonal and professional qualities.

Data from 240 students (85.5%) who completed peer assessment forms were analyzed. Peer assessed WH scores were predictive of later MSPE groups in second (F = 44.90,  < 0.001) and third years (F = 29.54,  < 0.001) while IA attributes were not predictive of these rankings. The program director surveys were completed on 43 students (44% response rate), and PD ratings significantly correlated with second ( = .32 [ = 0.015]) and third year ( = .43 [ = 0.004]) peer assessed WH scores but not with peer assessed IA scores.

Limitations of this study include being based at a single site and the low response rate of 44% from PDs which impacts the reliability and validity inferences of correlations between peer assessment and future ratings of graduates. In addition, this school has a well established infrastructure for conducting peer assessments; thus the importance of well-trained students for the task of rating their peers cannot be underestimated.

In conclusion, this study provides evidence to support peer assessments during medical school training. More specifically, identification of attributes related to work habits can be assessed by medical school peers and are predictive of later MSPE rankings and PD ratings. Such recognition, as early as during the second year of medical school, can be helpful to trigger remedial and educational interventions sooner rather than later.