What procedures are students doing during undergraduate surgical clerkship?

Abstract

Background

Many North American medical schools have removed didactic surgical teaching from the nonclinical years, and there has been a trend toward shortening surgical clerkships. Of concern is that this policy has led to a decrease in surgical exposure and a diminished interest in students pursuing a surgical career. We aimed to determine the effect of curricular change on practical experiences during surgical clerkship and to evaluate overall practical clinical exposure of students during surgical clerkship.

Methods

We collected validated experience logbooks completed before (1999–2001) and after (2001–2003) the curriculum change at the University of Alberta and converted them into electronic format. The study analyzed 10 procedures and 5 patient management situations. We assessed numbers of procedures performed and student performance on the Objective Structured Clinical Exam (OSCE) and Multiple-Choice Question (MCQ) examinations before and after the curriculum change. In addition, we completed an overall survey of all 4 classes (2000, 2001, 2002, 2003), measuring clinical exposure. We reviewed a total of 428 logbooks.

Results

There were significant gaps in clinical exposure, which was demonstrated by more than 70% of students in each class failing to complete 8 of 15 procedures or managements at least once. No significant change in practical surgical exposure resulted from the curriculum change. The curriculum change did result in a decrease in end-of-rotation MCQ score performance, which was demonstrated by a 5% decrease in the class average after the curriculum change. Students' performance on ward evaluations and their OSCE scores were unaffected.

Conclusion

We were encouraged that a major change in how surgical education is delivered did not have a detrimental effect on subjective and objective evaluations of student performance. However, we are concerned that a considerable number of students appeared to have not performed several inpatient procedures. Further study is warranted to determine whether this is a common problem in other schools. There is a clear need at our school, and no doubt at others, to establish skills centres and other strategies to ensure that this component of medical education is appropriately and effectively taught.

Surgical education is continually changing as new technologies and methodologies emerge. Owing to this state of change, it becomes necessary for medical schools to periodically re-evaluate core clinical competencies. Education delivery methods should also be periodically reviewed to determine the effectiveness of the clerkship program, as measured by standardized methods of evaluation (Multiple-Choice Question [MCQ] exams and Objective Structured Clinical Exams [OSCE]) and clinical exposure (measured through student logbooks).¹ In September of 2000, the University of Alberta implemented a major curriculum change. Prior to this, there was a 26–surgical lecture series in year 2, followed by 10 weeks of preceptor-and ward-based general and specialty surgery in year 4. With the curriculum change, the lecture series was replaced with 26 surgery topic seminars (i.e., 1–2 per day) during the surgery rotation. In addition, students now do 6 weeks of preceptor-and ward-based general surgery in year 3 and 6 weeks of ambulatory specialty surgery in year 4. Evaluative methods remained the same before and after the curriculum change. In addition to pre-and postrotation MCQ, faculty evaluation and postrotation OSCE, each student completed an experience log.² The logs recorded clinical encounters, surgical assists and procedures performed. Logbooks were to be completed by students during their rotation and evaluated by their preceptors at the end of the rotation; they provided up to 5% of the student's overall grade. We conducted this study to assess whether there were any deleterious effects of this curriculum change on surgical education and to gain insight into the type and quantity of procedures our medical students were actually performing during their core surgical rotation.

Methods

We retrospectively studied logbook entries of procedures and patient management protocols during the general surgery clerkship rotation.

We conducted this study at the University of Alberta, an accredited 4-year medical school, with approximately 120 medical students each year. The University of Alberta medical school uses 4 hospital sites for the surgical clerkship: the University Hospital, Grey Nuns Hospital, Misericordia Hospital and the Royal Alexandra Hospital. Each site performs daily elective and emergency surgeries, although the type and number of surgical cases differ among sites.

Study protocol

We collected and verified for completeness the logbooks of students graduating in the years 2000, 2001, 2002 and 2003 (n = 428). We excluded incomplete logbooks (n = 10) from the study. We developed an electronic database mimicking the format of the logbook. We converted the data into digital format and entered them into a spreadsheet, using Microsoft Excel version 2003. In addition, we collected OSCE scores, ward evaluations and MCQ test scores for 2000–2003.

Measurements and data analysis

We examined the following 15 procedures: resuscitation of a multiple trauma patient, resuscitation of an arrested patient, central line insertion, sigmoidoscopy, chest tube removal, arterial puncture, endotracheal intubation, nasogastric tube insertion, intravenous insertion, drain removal, foley catheter insertion, suturing, discussing malignancy with a patient, postoperative fever management and low urine output management. We determined the number of students who had completed a procedure or management at least once and compared this before and after the curriculum change. We also collected and compared OSCE scores, ward evaluations and MCQ scores, pre-and postcurriculum change. The same prerotation MCQ exam (n = 50 questions) was administered to each student before commencing the rotation. The postrotation MCQ exam (n = 50 questions) comprised questions derived from the University of Alberta multiple choice question bank and was administered at the end of each rotation, along with the OSCE.

Statistical analysis

We determined the number of times a procedure was performed by each student before the curriculum change and after the curriculum change. We compared the proportion of students who had performed a procedure at least once before and then after the curriculum change, using the chi-square test for independent proportions. We also calculated the average student scores on the faculty evaluation pre-and postrotation MCQ. We calculated the postrotation OSCE scores for the 2 periods and compared these, using the t test for 2 independent means. All tests were 2-tailed, and p values less than 0.05 were considered significant.

Results

The percentage of students who failed to perform a procedure at least once ranged between 1% and 92% over all 4 years examined. In 8 of 15 procedures, more than 70% of students failed to complete a procedure at least once. In 3 of 15 procedures, more than 80% of students failed to complete a procedure at least once (Table 1).

Table 1

After the curriculum change, the number of students who completed the following procedures at least once increased significantly: chest tube insertion/removal (p = 0.002), nasogastric tube insertion (p = 0.001) and endotracheal intubation (p = 0.001). The number of students who completed a chest tube insertion/removal at least once increased from 45 of 200 to 74 of 201, a 15% increase (Fig. 1a). The number of students who completed a nasogastric tube insertion at least once increased from 120 of 200 to 154, an increase of 17% (Fig. 1b). The number of students who completed an endotracheal intubation at least once increased from 48 of 200 to 83 of 201, an increase of 17% (Fig. 1c).

FIG. 1. The number of students who completed the following surgical procedures at least once before and after the curriculum change: A) chest tube insertion; (p = 0.002) B) nasogastruic tube insertion/removal (p = 0.001); C) endotracheal intubation (p = 0.001); D) resuscitation of an arrested patient (p = 0.004); and E) postoperative fever management (p = 0.013).

After the curriculum change, the number of students who completed the following procedures at least once decreased significantly: resuscitation of an arrested patient (p = 0.004) and postoperative fever management (p = 0.01). The number of students who completed a resuscitation of an arrested patient decreased from 38 to 18, a decrease of 10% (Fig. 1d). The number of students who managed a postoperative fever decreased from 174 to 155, a 10% decrease (Fig. 1e).

There was no significant change in OSCE or clerkship evaluations following the curriculum change. Prerotation MCQ exam scores decreased from an average mark of 57% to 51% after the curriculum change (Table 2). End-of-rotation MCQ exam scores also decreased from an average mark of 71% precurriculum change to 66% postcurriculum change (Table 2). These differences were not significant.

Table 2

Discussion

The transition from didactic teaching to clinical training is challenging and highlights the issue of what undergraduate medical students should be exposed to in order to be “clinically competent.” The questions concerning what procedures students are actually performing during their undergraduate surgical rotation and what procedures they should be competent in performing upon graduating medical school remain unresolved.³

The initial question, “What are students actually doing during their surgical rotation?” has been of interest for many years. There have been several attempts to monitor what students do, each having their own drawbacks. One such method of monitoring student activity is logbooks. Chatenay and colleagues² evaluated the efficacy of logbooks as a measure of student activity and effectively tracked the activities of students rotating through general surgery at the University of Alberta. Engum expanded on this model and digitized the logbooks into PDA (personal digital assistant) format and tracked the activities of undergraduate medical students at the Indiana School of Medicine.⁴ Both studies, although with minor limitations, accurately reported students' activities. Freeman and others⁵ used a surgical checklist to monitor students' activities in surgery and concluded that such monitoring is imperative to directing future educational initiatives.

Stemming from these and si-milar studies is the question of what procedures medical students should be proficient in performing upon entering residency. Forbes and colleagues³ identified a wide variation in the emphasis placed on basic learning objectives and perceived student performance. The author emphasized the need for national standards of competency to ensure that all graduates attain the minimal acceptable skill level. The logbooks used in this study were taken from Chatenay and colleagues'² study and included a wide variety of procedures, some of which are considered beyond the expectations of a medical student. These procedures were included to assess whether medical students were completing them and, thus, whether they should complete procedures traditionally considered to be at a resident level.

Establishing curricular objectives and periodic curricular evaluations (i.e., through logbooks and checklists) is imperative.¹ Many studies have been carried out that emphasize the importance of monitoring residents' activities and periodically evaluating whether program objectives are being met. Such an evaluation was documented at the University of Illinois, where postrotation surveys were used as a measure of skills and knowledge attained during the surgical rotation.⁶ This resulted in a curriculum change, designed to phase out dated objectives and provide more emphasis on areas of demand.

Similar to the study at the University of Illinois,⁶ our study results outline the strengths and deficiencies of undergraduate surgical education at the University of Alberta. An area of concern is the apparent lack of student participation in performing required procedures. There were many instances wherein most students in a class failed to complete a procedure at least once. One explanation for this is that clinical preceptors are not comfortable allowing a medical student to perform the required procedure. For example, 92% of students failed to be involved in the resuscitation of a multiple trauma patient. It is unrealistic to expect medical students at the University of Alberta to perform this procedure because they lack advanced trauma life support training, which is provided during the first year of residency training. Another potential source of discrepancy in the procedures performed is the disparity of experience that is inherent in a multisite (clinical teaching unit) program. Students being assigned to different sites within a city results in variation in case load, types of cases and quality of preceptor education.

The changes in the number of procedures performed before and after the curriculum change were negligible, with the exception of a few. This confirms that the curriculum change in 2001, which involved removing the surgical lecture series in year 2, moving the surgical clerkship from fourth to third year and shortening the duration of the rotation from 10 to 6 weeks, did not have a negative impact on students' overall clinical experience. This observation is supported by the fact that clinical ward evaluations and OSCE scores did not change markedly with the curriculum change; however, there was a decrease in class average in prerotation and postrotation multiple choice examination scores. This is to be expected, because part of the curriculum change involved eliminating a 24-lecture series from year 2 of the medical program. However, in support of this change, it was gratifying to see that, even with a shortened clerkship carried out a year earlier, postrotation average MCQ scores remained the same as those prior to the curriculum change.

The use of logbooks in this study provided insight into the activities of medical students rotating through general surgery. One limitation with using logbooks is the inherent bias of self-reporting, and students may not be sufficiently motivated to complete them. The logbooks were given a 5% weight in students' overall evaluation, but some students failed to complete their logs. Another concern is students fabricating entries to achieve a higher grade. The lack of reliability of student records limits the conclusions that can be drawn from this study. Increased weight on logbooks, as well as informing students of the use of these logbooks in educational research, could increase compliance and, in turn, solidify the preliminary results of this study.

Conclusion

This study identified significant gaps in surgical exposure, as demonstrated by more than one-half of the students failing to complete more than 50% of the procedures at least once. The decrease in the length of rotation from 8 weeks to 6 weeks had no effect on clinical exposure, and eliminating the 24-lecture series from year 2 of the MD program had negative effects on didactic evaluations but had no influence on OSCE scores and ward evaluations. The deficiencies in surgical exposure require further study. What procedures should medical students perform and what should be left to residency training? Does competency need to be demonstrated and evaluated, and should graduation be dependent on proof of performance? Are some procedures more important than others? Should medical students be trained in a dry laboratory in life-saving procedures that they are rarely exposed to during their clerkship, such as emergency tracheostomy? Further study is clearly required.

Presented at the Canadian Undergraduate Surgical Education Committee meeting in Whistler, 2003, and the Canadian Surgery Forum in Ottawa, 2004.

Competing interests: None declared.

Correspondence to: Dr. C.J. de Gara, Department of Surgical Oncology, Cross Cancer Institute, 11560 University Ave., Edmonton AB T6G 1Z2; fax 780 432-8333; chrisdeg@cancerboard.ab.ca