A retrospective review of general surgery training outcomes at the University of Toronto

Abstract

Background

Surgical educators have struggled with achieving an optimal balance between the service workload and education of surgical residents. In Ontario, a variety of factors during the past 12 years have had the net impact of reducing the clinical training experience of general surgery residents. We questioned what impact the reductions in trainee workload have had on general surgery graduates at the University of Toronto.

Methods

We evaluated graduates from the University of Toronto general surgery training program from 1995 to 2006. We compared final-year In-Training Evaluation Reports (ITERs) of trainees during this interval. For purposes of comparison, we subdivided residents into 4 groups according to year of graduation (1995–1997, 1998–2000, 2001–2003 and 2004–2006). We evaluated postgraduate “performance” by categorizing residents into 1 of 4 groups: first, residents who entered directly into general surgery practice after graduation; second, residents who entered into a certification subspecialty program of the Royal College of Physicians and Surgeons of Canada (RCPSC); third, residents who entered into a noncertification program of the RCPSC; and fourth, residents who entered into a variety of nonregulated “clinical fellowships.”

Results

We assessed and evaluated 118 of 134 surgical trainees (88%) in this study. We included in the study graduates for whom completed ITER records were available and postgraduate training records were known and validated. The mean scores for each of the 5 evaluated residency training parameters included in the ITER (technical skills, professional attitudes, application of knowledge, teaching performance and overall performance) were not statistically different for each of the 4 graduating groups from 1995 to 2006. However, we determined that there were statistically fewer general surgery graduates (p < 0.05) who entered directly into general surgery practice in the 2004–2006 group compared with the 1998–2000 and 2001–2003 groups. The graduates from 2004 to 2006 who did not enter into general surgery practice appeared to choose a clinical fellowship.

Conclusion

These observations may indicate that recent surgical graduates possess an acceptable skill set but may lack the clinical confidence and experience to enter directly into general surgery practice. Evidence seems to indicate that the clinical fellowship has become an unregulated surrogate extension of the training program whereby surgeons can gain additional clinical experience and surgical expertise.

Surgical educators have struggled with achieving an optimal balance between the service workload and education of surgical residents. In the past several decades, attention has primarily focused on reducing the work hours of trainees in an effort to improve resident well-being as well as patient safety.¹ Specifically, in Ontario the adoption and rigid application of the Professional Association of Internes and Residents of Ontario (PAIRO) and the Ontario Council of Teaching Hospitals (OCOTH) guidelines for resident work-hour thresholds has effectively reduced the workweek of general surgery residents.² Furthermore, in 1997, the Royal College of Physicians and Surgeons of Canada (RCPSC) introduced a change from fall to spring RCPSC certification examinations, which has essentially reduced the final several months of residency training to a period of written and oral exam preparation. To some extent, increased time constraints in the operating room as well as a heightened medico-legal awareness has also limited the independent operating experience of senior residents.³ Since the specialty of general surgery has arguably the broadest scope of practice of all the surgical disciplines, we questioned what impact the reductions in trainee workload have had on general surgery graduates at the University of Toronto. Specifically, we wished to determine how final-year resident performance, as judged by the In-Training Evaluation Report (ITER), changed during the past decade. We also wanted to determine whether the postgraduate career choices of surgical residents changed during the same interval. Specifically we questioned whether the proportion of graduates entering directly into general surgery practice significantly changed during this period.

Methods

We evaluated graduates from the University of Toronto general surgery training program from 1995 to 2006. We felt that this time frame incorporated many of the changes that have lead to a reduction in resident workload. We compared final-year ITER scores of trainees during this interval to determine whether there was any difference in clinical knowledge or technical ability of surgical residents. For purposes of comparison, we subdivided residents into 4 groups according to year of graduation (1995–1997, 1998–2000, 2001–2003 and 2004–2006). It was considered that this would best allow for intergroup comparisons, taking into consideration the possible influence of PAIRO–OCOTH contract agreements (1999, 2002) as well as the timing of the RCPSC change to spring examinations (1997). We evaluated postgraduate “performance” by categorizing residents into 1 of 4 groups. The first group included residents who entered directly into general surgery practice immediately after graduation. The second group included residents who entered into a subspecialty program that required an RCPSC certification examination, such as thoracic, vascular, pediatric and cardiac surgery, and intensive care medicine. A third group included residents who entered into a noncertification RCPSC program such as surgical oncology, colorectal surgery, hepatobiliary and pancreatic surgery, and head and neck surgery. The final group included residents who entered into a variety of “clinical fellowships,” which are not regulated by the RCPSC.

We analyzed the mean scores for the ITERs for intergroup differences using 1-way analysis of variance followed by Bonferroni and Scheffe multiple intergroup comparisons. We considered p < 0.05 to be statistically significant. We confidentially maintained all collected information using unique nonpersonal identifiers. Our methodology was approved by the Research Ethics Board at St. Joseph’s Health Centre, Toronto, Ont.

Results

We assessed and evaluated 118 of 134 surgical trainees (88%) in this study. We included in the study graduates for whom completed ITER records were available and postgraduate training records were known and validated. The mean scores for each of the 5 evaluated residency training parameters included in the ITER (technical skills, professional attitudes, application of knowledge, teaching performance and overall performance) were unchanged during the recorded time intervals (Table 1; F = 1.4, p > 0.05). Furthermore, the mean ITER scores were also statistically similar when evaluated according to postgraduate program choice (Table 2, F = 1.04, p > 0.05). Table 3 and Figure 1 reveal the percentages of each graduating group that chose either general surgery practice, an RCPSC sub-specialty program with certification examination, an RCPSC subspecialty program without certification examination, or a non-RCPSC clinical fellowship. There was a statistically significant reduction in the percentage of surgical graduates who chose to enter directly into general surgery practice from 2004 to 2006 versus graduates from 1998 to 2000, or 2001 to 2003 (p < 0.05). Although the comparison between graduates from 1995 to 1997 versus 2004 to 2006 was not statistically significant, it is worth noting that a relatively large number of general surgery graduates from 1995 to 1997 (12 or 54.5%) entered into an RCPSC certified training program. The graduates from 2004 to 2006 who did not enter into general surgery practice appeared to choose a clinical fellowship. These clinical fellowships were the most popular postgraduate choices for general surgery trainees during this 3-year interval.

Table 1.

Mean In-Training Evaluation Report scores of graduate general surgery residents according to year of graduation

Year of graduation	Training parameter, mean (SEM) score
	Technical skills	Professional attitudes	Application of knowledge	Teaching performance	Overall performance
1995–1997	4.15 (0.12)	4.38 (0.10)	4.00 (0.14)	4.22 (0.13)	4.10 (0.14)
1998–2000	4.15 (0.12)	4.38 (0.10)	4.00 (0.14)	4.22 (0.13)	4.10 (0.14)
2001–2003	4.32 (0.07)	4.46 (0.08)	4.21 (0.07)	4.24 (0.10)	4.23 (0.10)
2004–2006	4.21 (0.11)	4.37 (0.11)	4.29 (0.13)	4.24 (0.16)	4.36 (0.12)
p value	0.50	0.30	0.30	0.30	0.30

SEM = standard error of the mean.

Table 2.

Mean In-Training Evaluation Report scores of graduate general surgery residents according to postresidency program

Postresidency program	Training parameter, mean (SEM) score
	Technical skills	Professional attitudes	Application of knowledge	Teaching performance	Overall performance
General surgery practice	4.26 (0.10)	4.28 (0.13)	4.06 (0.12)	4.17 (0.13)	4.13 (0.11)
RCPSC subspecialty by exam	4.33 (0.07)	4.48 (0.08)	4.31 (0.08)	4.30 (0.09)	4.29 (0.08)
RCPSC subspecialty without exam	4.16 (0.14)	4.37 (0.12)	4.10 (0.16)	4.22 (0.16)	4.22 (0.15)
Non-RCPSC clinical fellowship	4.12 (0.08)	4.25 (0.09)	4.05 (0.09)	4.06 (0.11)	4.07 (0.10)
p value	0.30	0.30	0.10	0.60	0.40

RCPSC = Royal College of Physicians and Surgeons of Canada; SEM = standard error of the mean.

Table 3.

Postgraduate career choices of general surgery trainees at the University of Toronto

Postresidency program	Training period, no. (%) of trainees by year of graduation
	1995–1997	1998–2000	2001–2003	2004–2006	Total
General surgery practice	2 (9.1)	10 (29.4)	12 (30.8)	1 (4.3)*	25 (21.2)
RCPSC subspecialty by exam	12 (54.5)	11 (32.4)	14 (35.9)	6 (26.1)	43 (36.4)
RCPSC subspecialty without exam	3 (13.6)	2 (5.9)	4 (10.3)	6 (26.1)	15 (12.7)
Non-RCPSC clinical fellowship	5 (22.7)	11 (32.4)	9 (23.1)	10 (43.5)	35 (29.7)
Total	22 (100)	34 (100)	39 (100)	23 (100)	118 (100)

RCPSC = Royal College of Physicians and Surgeons of Canada.

^*p < 0.05 v. graduates entering into general surgery practice from 1998 to 2000 and from 2001 to 2003.

Fig. 1.

Distribution of general surgery residents’ postgraduate training, by year of graduation. *p = 0.04.

Discussion

For more than 20 years, both the medical community and the general public have been aware of the potential adverse effects that excessive work hours and fatigue in medical interns and residents have on clinical performance. The landmark case of Libby Zion at the New York Hospital in 1984 highlighted the potential implications of an exhausted resident’s actions in an 18-year-old woman’s death.⁴ Since this case was reported, a number of clinical research and observational studies have emphasized the need to monitor and limit the number of hours residents can work in order to maximize their education and personal health while ensuring a high quality of patient care. Landrigan and colleagues⁵ identified a 22% higher rate of serious errors in medical interns in an intensive care unit who completed “extended” 24-hour work shifts. Similarly, a recent study found that surgical residents made up to twice the number of technical errors in the performance of simulated laparoscopic surgical skills after working overnight versus after a night of sleep.^6,7

In Ontario, resident workload is contractually negotiated by PAIRO. Since its inception in 1974, PAIRO has been the official organization dedicated to improving the rights, interests and well-being of all training medical doctors in Ontario. One of PAIRO’s primary functions is the negotiation and enforcement of the collective agreement with OCOTH, which, among other things, legislates maximum duty hours and call hours for residents. The current agreement, which limits the maximum number of working hours, including on-call shifts, to about 72 hours per week, is applied uniformly to the diverse array of medical specialties offered in Ontario such as emergency medicine, anesthesia, psychiatry, pediatrics and neurosurgery.² PAIRO employs a system of random audits of call schedules to ensure that all teaching programs adhere to the collective agreement.

In addition to the influence of PAIRO, other factors have also affected the workload of general surgery trainees. Specifically, in 1997 the RCPSC changed the pre-existing fall certification examination (November) to a spring examination (June). This change was necessary to address the fact that surgical graduates otherwise would not have a general licence to practise medicine from June to November. The switch to a spring examination has, however, had an impact on surgical residents’ final 6 months of training. Specifically, in the weeks leading up to the fellowship written and oral examinations, residents are understandably “preoccupied” with exam preparation and less inclined to focus on their clinical education. Although the individual experience varies with each surgical trainee, the net result of this change in certification examination has been that the surgical educational and training experience of each final-year resident is diluted.

A focused interest in surgical training and resident work hours has not been limited to centres in Ontario or Canada. Internationally, many countries have recently modified surgical training programs and implemented work-hour restrictions. In 2003, the United States mandated the change from a 100+ hour resident workweek to an 80-hour per week standard.^8–11 In Denmark and Holland, surgical trainees currently have legislated schedules of 37.5 and 48 hours per week. Many educators would argue that a general surgery resident cannot be adequately trained in a 40-hour workweek time frame.^1,8,12,13 The scope of general surgery in particular, which entails an intimate knowledge of the anatomy and physiology of multiple organ systems, makes it the most complex and arguably the most challenging to master of the surgical disciplines. The Toronto general surgery training experience, at about 72 hours per week, would appear to strike a balance between the American and European systems. The optimum resident workload, within the construct of the 5-year general surgery training program, is currently unknown.

Resident ITER scores did not significantly change from 1995 to 2006 in our study. The ITER incorporates an evaluation of the technical skills, professional attitudes, clinical knowledge and teaching performance of each resident. The hypothesis that reduced resident workloads may lead to an overall reduction in clinical exposure and subsequent reduction in resident performance was not supported by our observations. The stability of these evaluation reports is not particularly surprising, however, given that scores are often administered to residents relative to their peers in a given year. In particular, the evaluation of resident technical skills on average did not significantly change during the 12-year observation period. In this context, the single dimension of observed technical skills of residents may not be the best measure of surgical ability.^14,15 A surgical trainee’s ability to tie suture and dissect tissue in an observed and supervised setting essentially isolates this technical exercise without addressing the broader question of the resident’s ability to synthesize a variety of clinical skills into practice. For example, surgical attributes that evolve with clinical experience, such as self-confidence, composure and clinical judgment, are not well represented in the ITER.¹⁶ Furthermore, in recent years the validity of these scaled standard ITERs in general have been questioned, as it is clear that faculty are generally reluctant to give low numerical scores.

In the past several years there has been a surprisingly small number of general surgery trainees from the University of Toronto who have chosen to enter into practice immediately after graduation. Only 1 graduate out of 23 in the past 3 years (4.3%) chose this career option. Meanwhile, 10 out of 23 graduates (43.5%) in the same interval proceeded with a variety of nonaccredited clinical fellowships after their RCPSC training. This observation marks a dramatic departure from the trend of previous graduating years. The current curriculum for this program as dictated by the RCPSC¹⁷ states that a competent resident in general surgery must achieve the following:

• knowledge and expertise in clinical and operative management of diseases of the alimentary tract, breast and endocrine systems, trauma and critical care, general surgical oncology and ambulatory patient care for general surgery;

• mastery of surgical skills of open cavitary surgery, endoscopy, minimal access surgery including abdominal wall surgery;

• effective clinical judgment and decision-making in dealing with general surgical problems based on sound surgical fundamentals.

This is undoubtedly an extensive list of proficiencies that must be acquired during a 5-year period. Furthermore, the RCPSC objectives state that “the specialty of general surgery embraces the principles of safe and effective surgical care of the whole person of any age, and is the parent of all surgical specialities.” Our retrospective observations may indicate that the current state of general surgical education is such that satisfying all RCPSC training objectives has become increasingly difficult. Furthermore, evidence seems to indicate that the clinical fellowship has become an unregulated surrogate extension of the training program whereby surgeons can gain additional clinical experience and surgical expertise.

The University of Toronto program is the largest general surgery training program in Canada. However, the results noted in our study may not be generalizable to all other residency centres. A unique aspect of surgical education at the University of Toronto is the Surgical Scientist Program, a highly regarded research training experience for surgical residents who wish to pursue a career in academic surgery. Under the guidance of an established scientist supervisor, residents interrupt their clinical training and commit to a minimum of 2 years of focused research activity. This Surgical Scientist Program, which was established in 1984, would likely attract surgical trainees with academic aspirations. However, to our knowledge there is no evidence to indicate that the overall composition of surgical trainees significantly changed during the 1995–2006 period of evaluation.

Admittedly there may be other explanations to account for the reduction in general surgery graduates immediately entering into practice. Several surgical subspecialty programs, such as surgical oncology and colorectal surgery, have grown and evolved during the study period. This is likely reflected in the increased number of postgraduate trainees (6 or 26.1%, Fig. 1) who entered into the noncertification RCPSC training stream from 2004 to 2006. Job availability does not seem to be an issue in deterring graduates from immediately establishing general surgery practices, as the job market in Ontario for general surgeons has been robust in the past decade with many senior staff retiring. Furthermore, the age and sex of graduate surgeons does not appear to be a factor. There may, however, be attitudinal factors influencing the observed trend. For example, current graduates may be less inclined to commit to a job offer at an institution before assessing opportunities elsewhere. In this context, many surgical graduates are carrying substantial debt and the prospect of incurring further debt in setting up an office may be understandably daunting. Of course, the 2004 to 2006 cohort may simply be anomalous and not reflect a trend away from general surgery graduates entering directly into surgical practice. Further observation and evaluation will be necessary to ascertain the significance of these observations.

It is important for surgical educators to be aware of and monitor outcomes within their training programs. These outcomes include not only ITER performance reviews and RCPSC licensing examination scores but also an assessment of postgraduate practice profiles. These postgraduate trends may ultimately reflect a need for change within the core training program. Invariably surgical educators need to provide an environment in which trainees can foster their skills, and residents must appreciate that a surgical apprenticeship requires a sufficient investment of time and energy to acquire these skills.