Abstract
Background
Although much attention has been paid to the role of deliberate practice as a means of achieving expert levels of performance in other medical specialties, little has been published regarding its role in maximizing orthopaedic surgery resident potential. As an initial step in this process, this study seeks to determine how residents and program directors (PDs) feel current time spent in training is allocated compared with a theoretical ideal distribution of time.
Questions/purposes
According to residents and PDs, (1) how do resident responsibilities change by level of training as perceived and idealized by residents and PDs? (2) How do resident and PD perceptions of current and ideal time distributions compare with one another? (3) Do the current training structures described by residents and PDs differ from what they feel represents an ideal time allocation construct that maximizes the educational value of residency training?
Methods
A survey was sent to orthopaedic surgery resident and PD members of the Midwest Orthopedic Surgical Skills Consortium asking how they felt residents’ time spent in training was distributed across 10 domains and four operating room (OR) roles and what they felt would be an ideal distribution of that time. Responses were compared between residents and PDs and between current schedules and ideal schedules.
Results
Both residents and PDs agreed that time currently spent in training differs by postgraduate year with senior-level residents spending more time in the OR (33.7% ± 8.3% versus 17.9% ± 6.2% [interns] and 27.4% ± 10.2% [juniors] according to residents, p < 0.001; and 38.6% ± 8.1% versus 11.8% ± 6.4% [interns] and 26.1% ± 5.7% [juniors] according to PD, p < 0.001). The same holds true for their theoretical ideals. Residents and PDs agree on current resident time allocation across the 10 domains; however, they disagree on multiple components of the ideal program with residents desiring more time spent in the OR than what PDs prefer (residents 40.3% ± 10.3% versus PD 32.6% ± 14.6% [mean difference {MD}, 7.7; 95% confidence interval {CI}, 4.4, 11.0], p < 0.001). Residents would also prefer to have more time spent deliberately practicing surgical skills outside of the OR (current 1.8% ± 2.1% versus ideal 3.7% ± 3.2% [MD, −1.9; 95% CI, −.2.4 to −1.4], p < 0.001). Both residents and PDs want residents to spend less time completing paperwork (current 4.4% ± 4.1% versus ideal 0.8% ± 1.6% [MD, 3.6; 95% CI, 3.0–4.2], p < 0.001 for residents; and current 3.6% ± 4.1% versus ideal 1.5% ± 1.9% [MD, 2.1; 95% CI, 0.9–3.3], p < 0.001 for PDs).
Conclusions
Residents and PDs seem to agree on how time is currently spent in residency training. Some differences of opinions continue to exist regarding how an ideal program should be structured; however, this work identifies a few potential targets for improvement that are agreed on by both residents and PDs. These areas include increasing OR time, finding opportunities for deliberate practice of surgical skills outside of the OR, and decreased clerical burden. This study may serve as a template to allow programs to continue to refine their educational models in an effort to achieve curricula that meet the desired goals of both learners and educators. Additionally, it is an initial step toward more objective identification of the optimal educational structure of an orthopaedic residency program.
Introduction
Within the last decade, many changes have been made to the structure of orthopaedic surgery residency training [15]. One of the most substantial changes was the 2003 reduction of duty hours to an average of 80 hours per week, which has led many orthopaedic residency programs to alter their curricula and training methodologies to allow residents to develop the skills necessary to become qualified orthopaedic surgeons while remaining compliant with the necessary regulatory guidelines [2, 13, 15].
Another important development taking place during this time is the heightened recognition of the role of deliberate practice in the attainment and maintenance of expert performance [5]. Ericsson et al have defined deliberate practice as “effortful activities designed to optimize improvement,” and it is felt to be a critical component to acquisition of expert performance [7]. Although the development of expertise had been studied extensively in disciplines such as athletics, music, and other arts for some time, it is only within the last few years that its value has become increasingly recognized within the medical community [5, 6]. Many fields such as internal medicine, emergency medicine, and general surgery have studied the importance of deliberate practice in transforming learners into experts, but little has been published regarding its role in creating expert orthopedic surgeons [4, 9–11].
If the goal of orthopaedic residency programs continues to be the graduation of qualified and skilled orthopaedic surgeons in this era of reduced work hours, then the overall quality of each of those hours is critically important. In other words, residency programs may benefit from maximizing time spent deliberately practicing the myriad of skills required to transform residents into proficient, possibly even expert, surgeons. As an initial step in attempting to improve residency training by increasing time allocated to activities that promote expert performance, it is important to first understand how current orthopaedic surgery residents actually spend their time during residency. Once this is established, the difference between current resident responsibilities and those of a theoretical ideal can be evaluated. Currently, data are lacking on what represents the ideal distribution of a resident’s time, and this study provides subjective opinions of what that ideal should look like according to residents and program directors (PDs). Although more objective assessments of optimal time allocation during residency training are warranted, this study may serve as an initial step in that process.
Study Questions
According to residents and PDs, (1) how do resident responsibilities change by level of training as perceived and idealized by residents and PDs? (2) How do resident and PD perceptions of current and ideal time distributions compare with one another? (3) Do the current training structures described by residents and PDs differ from what they feel represents an ideal time allocation construct that maximizes the educational value of residency training?
Materials and Methods
After approval of the institutional review board, an informal survey was conducted among residents to identify distinct domains that encompass the roles and responsibilities of orthopaedic surgery residents from intern to postgraduate year (PGY) 5. After contribution and review by over 20 residents, a construct consisting of 10 different and distinct domains was decided on (Table 1). These 10 domains are considered to be independent of one another with minimal overlap and are felt to encompass all pertinent tasks and practices that residents perform over the course of their training. These 10 domains should not in any way be confused with the six competency domains outlined in the American College of Graduate Medical Education Outcomes Project because they are completely separate entities [1]. Additionally, time spent in the operating room (OR) was subdivided into four distinct roles: primary surgeon, first assistant, second assistant, and independent observer (not scrubbed in) (Table 2).
Table 1.
Domains of resident responsibilities
| 1 | Operating or performing procedures |
| 2 | Evaluating patients in a clinical setting |
| 3 | Caring for inpatients in the hospital |
| 4 | Evaluating patients in an acute care/emergency department setting |
| 5 | Independent studying or preparing for surgery/clinic/lectures |
| 6 | Attending formal lectures/courses |
| 7 | Answering patient phone calls/messages/questions/etc outside of the clinical setting |
| 8 | Completing paperwork (prescriptions, disability, insurance forms, duty hours, etc) |
| 9 | Conducting clinical or basic science research |
| 10 | Deliberate practice of surgical skills outside of the operating room setting |
| 11 | Other |
Table 2.
Resident roles in the operating room
| 1 | Primary surgeon (making surgical decisions and performing the operation) |
| 2 | First assistant |
| 3 | Second assistant |
| 4 | Observation only (not scrubbed in) |
Using these 10 domains and four OR roles, an electronic survey was created (Qualtrics software, http://www.qualtrics.com) and distributed by email to all residents (n = 353) and PDs (n = 11) of the 11 charter programs of the Midwest Orthopedic Surgical Skills Consortium (MOSSC) (Table 3). Respondents were given 3 months to complete the survey in its entirety, and reminders were sent after 1 and 2 months. Although residents were asked to provide their PGY, no other identifying information was requested to maintain appropriate anonymity.
Table 3.
Orthopaedic residency programs belonging to Midwest Orthopaedic Surgical Skills Consortium participating programs
| Dartmouth-Hitchcock Medical Center |
| University of Iowa Hospitals and Clinics |
| Mayo Clinic College of Medicine |
| University of Missouri-Columbia Program |
| University of Nebraska Medical Center College of Medicine |
| McGaw Medical Center of Northwestern University |
| Rush University Medical Center Program |
| St Louis University School of Medicine |
| University of Minnesota |
| Washington University in St Louis |
| University of Wisconsin |
The first portion of the survey focused on determining the perceived percentage of residents’ time during their current residency training that was allocated to each of the 10 defined domains. An “other” category was included to permit allocation of time to any activity not encompassed within the 10 domains. This question was followed by a description of how residents’ current time spent in surgery was distributed across the four OR roles. The second part of the survey then asked residents to describe how they would allocate time to each of the 10 domains and four OR roles in what they considered to be an “ideal” residency program. For each portion of the survey, respondents were asked to provide answers that added up to 100%. A running total was provided, and respondents were not allowed to advance to the next question until all values added up to exactly 100%. To increase precision, whole number values were not required, and respondents could enter fractional percentages if they desired.
The survey sent to PDs was nearly identical to the one sent to residents; however, PDs were asked to fill in current and ideal resident time allocation values for their current residents. They were asked to provide separate answers for interns (PGY-1), junior-level residents (PGY-2 and -3), and senior-level residents (PGY-4 and -5). For each resident level (intern, junior, and senior), running totals were provided and PD answers had to add up to 100% before they were permitted to progress to the next question.
Survey data were collected for each program and training level. For data analysis, the following subgroups were created for the resident responses: intern (PGY-1), junior-level resident (PGY-2 and -3), senior-level resident (PGY-4 and -5), and all residents. The same four subgroups were created for the responses of PDs. Resident and PD responses were compared with one another across the three subgroups of interns, juniors, and seniors. The cumulative responses of all residents were compared with the cumulative responses of PDs with each PGY receiving equal weight. After comparison between groups was completed, comparison within groups was made by contrasting residents’ opinions of their current roles with their ideal allocation of time across the domains and OR roles. PDs’ opinions of current roles were also compared with their ideal time construct for the 10 distinct domains and the four OR roles.
The survey was distributed to a total of 353 residents and 11 PDs. It was completed by 231 residents (65%) and all 11 PDs (100%) for an overall response rate of 67%. Response rates varied by individual program, ranging from a low of 30% to a high of 100% for the 11 programs analyzed. Interns demonstrated a higher response rate (57 of 71 [80%]) compared with PGY-2s (42 of 71 [59%]), PGY-3s (44 of 71 [62%]), PGY-4s (44 of 70 [63%]), and PGY-5s (44 of 70 [63%]).
When comparing means of two groups, variables were considered to be continuous and normally distributed, and a Student’s t-test was used to determine statistical significance (ie, all residents to PDs). Analysis of variance was used when comparing means of normally distributed continuous variables among three groups (ie, interns to juniors to seniors). Although a p value of < 0.05 was initially considered to be potentially significant for all analyses, a Bonferroni correction was applied to reduce the risk of committing a type I error (accepting false-positives), and only p values < 0.001 were considered significant. Because a total of 56 pairwise comparisons were made for this study (14 variables compared four ways), 0.05 was divided by 56 to achieve a Bonferroni adjusted p value of 0.0008 (which was rounded to 0.001).
Results
How Do Resident Responsibilities Change by Level of Training as Perceived and Idealized by Residents and Program Directors?
When comparing residents’ perceptions of the allocation of their actual time spent in training, differences exist among interns, junior residents, and senior residents (Table 4). Residents feel that the time they currently spend in the OR, clinic, inpatient setting, and acute care/emergency room differs based on PGY with seniors spending more time in the OR (33.7% ± 8.3% versus 17.9% ± 6.2% [interns] and 27.4% ± 10.2% [juniors], p < 0.001) and clinic (25.5% ± 7.2% versus 19.4% ± 7.9% [interns] and 23.6% ± 7.5% [juniors], p < 0.001) and lower-level residents spending more time caring for hospitalized (26.2% ± 13.9% versus 11.2% ± 5.1% [juniors] and 9.0% ± 3.9% [seniors], p < 0.001) and acute care patients (9.4% ± 5.1% versus 10.4% ± 5.6% [juniors] and 4.9% ± 3.3% [seniors], p < 0.001). Differences also exist for their current roles in the OR with senior-level residents spending more time serving as the primary surgeon (38.0% ± 18.8% versus 1.5% ± 3.6% [interns] and 19.9% ± 19.2% [juniors], p < 0.001) and interns spending the majority of their time as second assistants (65.4% ± 18.2% versus 20.3% ± 20.5% [juniors] and 8.2% ± 9.5% [seniors], p < 0.001). This discrepancy in current time allocation is similar to the discrepancies noted for residents’ ideal time allocation construct (Table 4). In this ideal resident training model, seniors desire more time spent in the OR (43.3% ± 9.9% versus 33.9% ± 10.2% [interns] and 41.3% ± 10.9% [juniors], p < 0.001) and less time in the inpatient (6.8% ± 3.5% versus 12.3% ± 7.5% [interns] and 7.6% ± 3.4% [juniors], p < 0.001) and acute care settings (5.8% ± 3.6% versus 10.5% ± 5.1% [interns] and 8.3% ± 3.6% [juniors], p < 0.001) compared with junior-level residents and interns. Additionally, they desire less time serving as first (36.6% ± 17.1% versus 52.5% ± 21.0% [interns] and 46.0% ± 21.2% [juniors], p < 0.001) and second assistants (4.2% ± 6.8% versus 17.5% ± 19.7% [interns] and 4.9% ± 8.4% [juniors], p < 0.001) and more time as primary surgeons (58.1% ± 17.5% versus 29.0% ± 22.1% [interns] versus 48.0% ± 22.2% [juniors], p < 0.001). Analysis of PDs’ perceptions of how their residents’ current time is allocated also differs by level of seniority (Table 5). Currently, they feel that senior-level residents spend more time operating (38.6% ± 8.1% versus 11.8% ± 6.4% [interns] and 26.1% ± 5.7% [juniors], p < 0.001) and less time caring for inpatients (10.0% ± 6.7% versus 26.8% ± 12.7% [interns] versus 13.9% ± 6.8% [juniors], p < 0.001) than junior residents and interns. PDs also feel that their interns and juniors are spending less time as the primary surgeon in the OR (5.6% ± 14.9% [interns] and 22.3% ± 20.2% [juniors] versus 60.0% ± 18.8% [seniors], p < 0.001) and more time as second assistants (57.3% ± 23.2% [interns] and 15.5% ± 10.6% [juniors] versus 2.7% ± 6.1% [seniors], p < 0.001). When asked to create an ideal schedule for their residents, PDs allocated similar amounts of time for each of the 10 domains with the exception of time spent in the OR (16.8% ± 10.8% [interns] versus 30.9% ± 9.2% [juniors] versus 42.1% ± 11.3% [seniors], p < 0.001). In the OR, they felt that senior-level residents should spend more time as the primary surgeon (72.7% ± 16.2% versus 8.2% ± 15.9% [interns] and 31.4% ± 19.3% [juniors], p < 0.001) and less time as a second assistant (1.4% ± 3.2% versus 43.6% ± 27.6% [interns] and 10.9% ± 12.8% [juniors], p < 0.001).
Table 4.
Comparison of resident views across all residents
| Resident views | Mean percent of time allocated to domain | p value* | |||
|---|---|---|---|---|---|
| Interns (SD) (n = 57) | Juniors (SD) (n = 86) | Seniors (SD) (n = 88) | |||
| Current resident responsibilities | |||||
| 1 | Operating or performing procedures | 17.9 (6.2) | 27.4 (10.2) | 33.7 (8.3) | < 0.001† |
| 2 | Evaluating patients in a clinical setting | 19.4 (7.9) | 23.6 (7.5) | 25.5 (7.2) | < 0.001† |
| 3 | Caring for inpatients in the hospital | 26.1 (13.9) | 11.2 (5.1) | 9.0 (3.9) | < 0.001† |
| 4 | Evaluating patients in an acute care/emergency department setting | 9.4 (5.1) | 10.4 (5.0) | 4.9 (3.3) | < 0.001† |
| 5 | Independent studying or preparing for surgery/clinic/lectures | 7.6 (4.0) | 8.1 (4.5) | 8.4 (5.1) | 0.602 |
| 6 | Attending formal lectures/courses | 5.4 (3.4) | 5.6 (2.7) | 5.1 (2.8) | 0.602 |
| 7 | Answering patient phone calls/messages/questions/etc outside of clinical setting | 2.5 (2.4) | 3.5 (3.5) | 3.3 (3.3) | 0.122 |
| 8 | Completing paperwork (prescriptions, disability, insurance forms, duty hours, etc) | 4.8 (4.1) | 4.6 (5.0) | 4.0 (3.2) | 0.481 |
| 9 | Conducting clinical or basic science research | 3.1 (2.4) | 3.4 (2.5) | 4.3 (3.5) | 0.043 |
| 10 | Deliberate practice of surgical skills outside of the operating room setting | 2.3 (2.2) | 1.6 (2.1) | 1.6 (2.0) | 0.077 |
| 11 | Other | 1.3 (3.8) | 0.7 (2.8) | 0.3 (1.7) | 0.073 |
| Current resident roles in the operating room | |||||
| 1 | Primary surgeon (making surgical decisions and performing the operation) | 1.5 (3.6) | 19.9 (19.2) | 38.0 (18.8) | < 0.001† |
| 2 | First assistant | 28.2 (17.0) | 58.2 (20.5) | 52.6 (18.4) | < 0.001† |
| 3 | Second assistant | 65.4 (18.2) | 20.3 (20.5) | 8.2 (9.5) | < 0.001† |
| 4 | Observation only (not scrubbed in) | 4.9 (9.7) | 1.6 (3.0) | 1.3 (2.7) | < 0.001† |
| Ideal resident responsibilities | |||||
| 1 | Operating or performing procedures | 33.9 (10.2) | 41.3 (10.9) | 43.4 (9.9) | < 0.001† |
| 2 | Evaluating patients in a clinical setting | 18.4 (6.8) | 19.6 (7.2) | 21.9 (6.8) | 0.008 |
| 3 | Caring for inpatients in the hospital | 12.3 (7.5) | 7.6 (3.4) | 6.8 (3.5) | < 0.001† |
| 4 | Evaluating patients in an acute care/emergency department setting | 10.6 (5.1) | 8.3 (3.6) | 5.8 (3.6) | < 0.001† |
| 5 | Independent studying or preparing for surgery/clinic/lectures | 8.0 (4.0) | 8.3 (4.4) | 7.7 (3.9) | 0.673 |
| 6 | Attending formal lectures/courses | 5.5 (2.7) | 5.2 (2.7) | 4.7 (2.6) | 0.194 |
| 7 | Answering patient phone calls/messages/questions/etc outside of clinical setting | 1.0 (1.92) | 0.7 (1.6) | 0.6 (1.0) | 0.232 |
| 8 | Completing paperwork (prescriptions, disability, insurance forms, duty hours, etc) | 1.1 (1.8) | 0.7 (1.6) | 0.9 (1.5) | 0.352 |
| 9 | Conducting clinical or basic science research | 4.9 (2.8) | 4.6 (3.7) | 4.9 (3.6) | 0.792 |
| 10 | Deliberate practice of surgical skills outside of the operating room setting | 4.4 (3.5) | 3.6 (3.2) | 3.3 (3.0) | 0.105 |
| 11 | Other | 0.1 (0.7) | 0.1 (0.7) | 0.0 (0.0) | 0.292 |
| Ideal resident roles in the operating room | |||||
| 1 | Primary surgeon (making surgical decisions and performing the operation) | 29.0 (22.1) | 48.0(22.2) | 58.1 (17.5) | < 0.001† |
| 2 | First assistant | 52.5 (21.0) | 46.0 (21.2) | 36.6 (17.1) | < 0.001† |
| 3 | Second assistant | 17.5 (19.7) | 4.9 (8.4) | 4.2 (6.8) | < 0.001† |
| 4 | Observation only (not scrubbed in) | 1.1 (4.0) | 1.1 (6.2) | 1.0 (2.7) | 1.000 |
* Comparisons made across all groups using analysis of variance; †only p values < 0.001 are considered to represent statistical significance.
Table 5.
Comparison of program director views across all residents
| Program director views | Mean percent of time allocated to domain | p value* | |||
|---|---|---|---|---|---|
| Interns (SD) (n = 11) | Juniors (SD) (n = 22) | Seniors (SD) (n = 22) | |||
| Current resident responsibilities | |||||
| 1 | Operating or performing procedures | 11.8 (6.4) | 26.1 (5.7) | 38.6 (8.1) | < 0.001† |
| 2 | Evaluating patients in a clinical setting | 20.9 (8.9) | 21.0 (7.9) | 21.4 (10.3) | 0.992 |
| 3 | Caring for inpatients in the hospital | 26.8 (12.7) | 13.9 (6.8) | 10.0 (6.7) | < 0.001† |
| 4 | Evaluating patients in an acute care/emergency department setting | 11.4 (5.5) | 12.3 (7.5) | 4.3 (3.6) | 0.005 |
| 5 | Independent studying or preparing for surgery/clinic/lectures | 7.8 (6.0) | 7.7 (5.6) | 9.0 (5.0) | 0.836 |
| 6 | Attending formal lectures/courses | 4.7 (2.4) | 5.2 (2.9) | 4.7 (2.4) | 0.895 |
| 7 | Answering patient phone calls/messages/questions/etc outside of clinical setting | 2.7 (2.4) | 3.3 (3.3) | 1.9 (3.1) | 0.562 |
| 8 | Completing paperwork (prescriptions, disability, insurance forms, duty hours, etc) | 6.4 (5.5) | 2.8 (3.2) | 3.1 (2.1) | 0.069 |
| 9 | Conducting clinical or basic science research | 1.7 (2.3) | 5.8 (6.0) | 5.2 (3.3) | 0.063 |
| 10 | Deliberate practice of surgical skills outside of the operating room setting | 5.7 (3.7) | 1.8 (2.7) | 1.6 (2.5) | 0.004 |
| 11 | Other | 0.0 (0.0) | 0.3 (0.9) | 0.3 (0.9) | 0.614 |
| Current resident roles in the operating room | |||||
| 1 | Primary surgeon (making surgical decisions and performing the operation) | 5.6 (14.9) | 22.3 (20.2) | 60.0 (18.8) | < 0.001† |
| 2 | First assistant | 30.0 (14.0) | 60.9 (19.2) | 36.8 (17.1) | < 0.001† |
| 3 | Second assistant | 57.3 (23.2) | 15.5 (10.6) | 2.7 (6.1) | < 0.001† |
| 4 | Observation only (not scrubbed in) | 7.2 (15.0) | 1.4 (3.2) | 0.5 (1.5) | 0.175 |
| Ideal resident responsibilities | |||||
| 1 | Operating or performing procedures | 16.8 (10.8) | 30.9 (9.2) | 42.1 (11.3) | < 0.001† |
| 2 | Evaluating patients in a clinical setting | 27.0 (14.7) | 21.8 (10.6) | 22.1 (11.8) | 0.552 |
| 3 | Caring for inpatients in the hospital | 18.6 (9.0) | 11.4 (6.7) | 9.4 (6.3) | 0.016 |
| 4 | Evaluating patients in an acute care/emergency department setting | 11.8 (5.1) | 10.5 (5.7) | 6.1 (3.0) | 0.022 |
| 5 | Independent studying or preparing for surgery/clinic/lectures | 6.7 (7.2) | 7.0 (5.4) | 6.9 (6.4) | 0.995 |
| 6 | Attending formal lectures/courses | 3.5 (2.2) | 4.0 (2.9) | 3.6 (2.1) | 0.851 |
| 7 | Answering patient phone calls/messages/questions/etc outside of clinical setting | 0.9 (1.6) | 1.4 (2.0) | 1.6 (2.3) | 0.687 |
| 8 | Completing paperwork (prescriptions, disability, insurance forms, duty hours, etc) | 0.8 (1.5) | 1.4 (1.9) | 2.0 (2.1) | 0.383 |
| 9 | Conducting clinical or basic science research | 5.6 (6.8) | 7.3 (6.3) | 4.2 (4.0) | 0.469 |
| 10 | Deliberate practice of surgical skills outside of the operating room setting | 7.8 (7.1) | 4.2 (3.1) | 1.9 (2.5) | 0.021 |
| 11 | Other | 0.5 (1.5) | 0.2 (0.6) | 0.3 (0.9) | 0.836 |
| Ideal resident roles in the operating room | |||||
| 1 | Primary surgeon (making surgical decisions and performing the operation) | 8.2 (15.9) | 31.4 (19.3) | 72.7 (16.2) | < 0.001† |
| 2 | First assistant | 42.3 (23.8) | 54.1 (13.4) | 24.9 (14.8) | 0.003 |
| 3 | Second assistant | 43.6 (27.6) | 10.9 (12.8) | 1.4 (3.2) | < 0.001† |
| 4 | Observation only (not scrubbed in) | 5.9 (9.7) | 0.9 (3.0) | 1.0 (3.0) | 0.107 |
* Comparisons made across all groups using analysis of variance; †only p values < 0.001 are considered to represent statistical significance.
How Do Resident and Program Director Perceptions of Current and Ideal Time Distributions Compare With One Another?
Overall, residents and PDs agreed on how residents’ current time is spent in training for each of the 10 domains when comparing all resident responses with all PD responses (Table 6). Regarding the four roles in the OR, the only discrepancy between PD perceptions and that of residents was that PDs felt residents were spending 34.0% ± 29.0% of their time acting as primary surgeons, whereas residents felt they were spending 22.3% ± 16.3% of their time in that role (mean difference [MD], −11.8; 95% confidence interval [CI], −17.5 to −6.0, p < 0.001). Multiple differences were noted when comparing the ideal program time construct of residents with that of the PDs. Residents desired 40.3% ± 10.3% of their time dedicated to the OR, whereas PDs prefer 32.6% ± 14.6% (MD, 7.7; 95% CI, 4.4–11.0, p < 0.001). Residents also prefer less time in the inpatient setting than PDs would like for them to have (8.46% ± 4.8% versus 12.0% ± 8.3% [MD, −3.6; 95% CI, −5.2 to −1.9], p < 0.001). The only other discrepancy noted was that residents would prefer to have more time attending formal lectures and courses than PDs feel is optimal (5.1% ± 2.7% versus 3.7% ± 2.4% [MD, 1.4; 95% CI, 0.6–2.1], p < 0.001). No significant differences existed for the ideal time allocated to in each of the four roles in the OR.
Table 6.
Perceptions of residents compared with program directors
| Residents’ perceptions compared with program directors | Mean percent of time allocated to domain | 95% confidence interval | p value* | |||
|---|---|---|---|---|---|---|
| Residents (SD) (n = 231) | Program directors (SD) (n = 55) | Mean difference | ||||
| Current resident responsibilities | ||||||
| 1 | Operating or performing procedures | 27.4 (8.5) | 28.3 (12.9) | −0.8 | −3.63 to 2.01 | 0.572 |
| 2 | Evaluating patients in a clinical setting | 23.3 (7.5) | 21.1 (8.8) | 2.2 | −0.13 to 4.43 | 0.065 |
| 3 | Caring for inpatients in the hospital | 14.0 (8.1) | 14.9 (11.5) | −0.9 | −3.50 to 1.72 | 0.503 |
| 4 | Evaluating patients in an acute care/emergency department setting | 8.1 (4.4) | 8.9 (6.7) | −0.8 | −2.30 to 0.62 | 0.257 |
| 5 | Independent studying or preparing for surgery/clinic/lectures | 8.1 (4.6) | 8.3 (5.4) | −0.2 | −1.55 to 1.25 | 0.833 |
| 6 | Attending formal lectures/courses | 5.4 (2.9) | 4.9 (2.5) | 0.5 | −0.39 to 1.29 | 0.293 |
| 7 | Answering patient phone calls/messages/questions/etc outside of clinical setting | 3.2 (3.1) | 2.6 (2.9) | 0.6 | −0.34 to 1.48 | 0.220 |
| 8 | Completing paperwork (prescriptions, disability, insurance forms, duty hours, etc) | 4.4 (4.1) | 3.6 (4.1) | 0.8 | −0.40 to 2.04 | 0.187 |
| 9 | Conducting clinical or basic science research | 3.7 (2.9) | 4.8 (4.4) | −1.1 | −2.03 to −0.13 | 0.027 |
| 10 | Deliberate practice of surgical skills outside of the operating room setting | 1.8 (2.1) | 2.5 (3.5) | −0.7 | −1.39 to 0.03 | 0.062 |
| 11 | Other | 0.7 (2.8) | 0.2 (0.7) | 0.5 | −0.28 to 1.20 | 0.225 |
| Current resident roles in the operating room | ||||||
| 1 | Primary surgeon (making surgical decisions and performing the operation) | 22.3 (16.3) | 34.0 (29.0) | −11.8 | −17.48 to −6.04 | < 0.001† |
| 2 | First assistant | 48.7 (18.7) | 45.1 (21.2) | 3.6 | −2.10 to 9.26 | 0.216 |
| 3 | Second assistant | 26.8 (16.5) | 18.7 (27.8) | 8.1 | 2.41−13.75 | 0.005 |
| 4 | Observation only (not scrubbed in) | 2.3 (5.5) | 2.2 (9.1) | 0.1 | −1.77 to 1.99 | 0.909 |
| Ideal resident responsibilities | ||||||
| 1 | Operating or performing procedures | 40.3 (10.2) | 32.6 (14.6) | 7.7 | 4.42−11.04 | < 0.001† |
| 2 | Evaluating patients in a clinical setting | 20.2 (6.9) | 23.0 (12.3) | −2.8 | −5.19 to −0.33 | 2.026 |
| 3 | Caring for inpatients in the hospital | 8.5 (4.8) | 12.0 (8.3) | −3.6 | −5.22 to −1.90 | < 0.001† |
| 4 | Evaluating patients in an acute care/emergency department setting | 7.9 (4.1) | 9.0 (5.2) | −1.1 | −2.35 to 0.19 | 0.095 |
| 5 | Independent studying or preparing for surgery/clinic/lectures | 8.0 (4.1) | 6.9 (6.2) | 1.1 | −0.26 to 2.44 | 0.114 |
| 6 | Attending formal lectures/courses | 5.1 (2.6) | 3.7 (2.4) | 1.4 | 0.60−2.14 | < 0.001† |
| 7 | Answering patient phone calls/messages/questions/etc outside of clinical setting | 0.7 (1.5) | 1.4 (1.9) | −0.6 | 1.12 to −0.18 | 0.007 |
| 8 | Completing paperwork prescriptions, disability, insurance forms, duty hours, etc) | 0.8 (1.6) | 1.5 (1.9) | −0.7 | −1.16 to −0.18 | 0.008 |
| 9 | Conducting clinical or basic science research | 4.8 (3.4) | 5.7 (5.8) | −1.0 | −2.15 to 0.21 | 0.106 |
| 10 | Deliberate practice of surgical skills outside of the operating room setting | 3.7 (3.2) | 4.0 (5.2) | −0.3 | −1.40 to 0.76 | 0.560 |
| 11 | Other | 0.1 (0.6) | 0.3 (1.1) | −0.2 | −0.39 to 0.01 | 0.061 |
| Ideal resident roles in the operating room | ||||||
| 1 | Primary surgeon making surgical decisions and performing the operation) | 47.1 (20.4) | 43.3 (31.8) | 3.9 | −2.93 to 10.67 | 0.264 |
| 2 | First assistant | 44.1 (19.6) | 40.1 (21.2) | 4.0 | −1.88 to 9.90 | 0.181 |
| 3 | Second assistant | 7.8 (12.0) | 13.6 (25.1) | −5.9 | −10.43 to −1.35 | 0.011 |
| 4 | Observation only (not scrubbed in) | 1.1 (4.6) | 2.0 (6.4) | −0.9 | −2.37 to 0.57 | 0.229 |
* Comparisons made between means of residents and program directors using t-test; †only p values < 0.001 are considered to represent statistical significance.
Do the Current Training Structures Described by Residents and Program Directors Differ From What They Feel Represents an Ideal Time Allocation Construct That Maximizes the Educational Value of Residency Training?
Comparison of resident perceptions of their current roles and responsibilities to their ideal time construct reveals discrepancies for seven of the 10 (70%) domains and two of the four (50%) OR roles (Table 7). Residents desire more time operating (current 27.4% ± 8.5% versus ideal 40.3% ± 10.3% [MD, −12.9; 95% CI, 14.6 to −11.1], p < 0.001), conducting clinical or basic science research (current 3.7% ± 2.9% versus ideal 4.8% ± 3.4% [MD, −1.1; 95% CI, −1.67 to −0.51], p < 0.001), and deliberately practicing surgical skills outside of the OR setting (current 1.8% ± 2.1% versus ideal 3.7% ± 3.2% [MD, −1.9; 95% CI, −2.4 to −1.4], p < 0.001). The domains that they desire less time allocated to include: evaluating patients in the clinic (current 23.3% ± 7.5% versus ideal 20.2% ± 6.2% [MD, 3.1; 95% CI, 1.8−4.4], p < 0.001), caring for inpatients (current 14.0% ± 8.1% versus ideal 8.5% ± 4.8% [MD, 5.6; 95% CI, 4.4–6.8], p < 0.001), responding to patient phone calls (current 3.2% ± 3.1% versus ideal 0.7% ± 1.5% [MD, 2.5; 95% CI, 2.0–2.9], p < 0.001), and completing paperwork (current 4.4% ± 4.1% versus ideal 0.8% ± 1.6% [MD, 3.6; 95% CI, 3.0–4.2], p < 0.001). In the OR, residents would prefer more time spent as the primary surgeon (current 22.3% ± 16.3% versus ideal 47.1% ± 20.4% [MD, −24.9; 95% CI, −28.3 to −21.5], p < 0.001) and less time spent as second assistants (current 26.8% ± 16.5% versus ideal 7.8% ± 12.0% [MD, 19.1; 95% CI, 16.4–21.7], p < 0.001) than what they are currently doing. When making the same comparison of the PDs’ perception of residents’ current roles to their idea of an ideal schedule, PDs only felt that there was one discrepancy among the 10 domains (10%) and no discrepancies in the OR roles (Table 8). Ideally, PDs desire that their residents spend less time completing paperwork (current 3.6% ± 4.1% versus ideal 1.5% ± 1.9% [MD, 2.1; 95% CI, 0.9–3.3], p < 0.001) than what they are currently doing.
Table 7.
Resident perceptions of their current time allocation versus their ideal
| Resident perceptions | Mean percent of time allocated to domain | 95% confidence interval | p value* | |||
|---|---|---|---|---|---|---|
| Residents’ perception of current time allocation (SD) (n = 231) | Residents’ perception of ideal time allocation (SD) (n = 231) | Mean difference | ||||
| Domains of resident responsibilities | ||||||
| 1 | Operating or performing procedures | 27.4 (8.5) | 40.3 (10.3) | −12.8 | −14.57 to −11.11 | < 0.001† |
| 2 | Evaluating patients in a clinical setting | 23.3 (7.5) | 20.2 (6.9) | 3.1 | 1.76–4.38 | < 0.001† |
| 3 | Caring for inpatients in the hospital | 14.0 (8.1) | 8.5 (4.8) | 5.6 | 4.36–6.80 | < 0.001† |
| 4 | Evaluating patients in an acute care/emergency department setting | 8.1 (4.4) | 7.9 (4.1) | 0.2 | −0.63 to 0.93 | 0.704 |
| 5 | Independent studying or preparing for surgery/clinic/lectures | 8.1 (4.6) | 8.0 (4.1) | 0.1 | −0.70 to 0.90 | 0.805 |
| 6 | Attending formal lectures/courses | 5.4 (2.9) | 5.1 (2.7) | 0.3 | −0.23 to 0.79 | 0.281 |
| 7 | Answering patient phone calls/messages/questions/etc outside of clinical setting | 3.2 (3.1) | 0.7 (1.5) | 2.5 | 2.01–2.91 | < 0.001† |
| 8 | Completing paperwork (prescriptions, disability, insurance forms, duty hours, etc) | 4.4 (4.1) | 0.8 (1.6) | 3.6 | 3.01–4.15 | < 0.001† |
| 9 | Conducting clinical or basic science research | 3.7 (2.9) | 4.8 (3.4) | −1.1 | −1.67 to −0.51 | < 0.001† |
| 10 | Deliberate practice of surgical skills outside of the operating room setting | 1.8 (2.1) | 3.7 (3.2) | −1.9 | −2.38 to −1.40 | < 0.001† |
| 11 | Other | 0.7 (2.8) | 0.1 (0.6) | 0.6 | 0.23–0.97 | 1.001 |
| Current resident roles in the operating room | ||||||
| 1 | Primary surgeon (making surgical decisions and performing the operation) | 22.3 (16.3) | 47.1 (20.4) | −24.9 | −28.29 to −21.53 | < 0.001† |
| 2 | First assistant | 48.7 (18.7) | 44.1 (19.6) | 4.6 | 1.10–8.12 | 1.010 |
| 3 | Second assistant | 26.8 (16.5) | 7.8 (12.0) | 19.1 | 16.42–21.70 | < 0.001† |
| 4 | Observation only (not scrubbed in) | 2.3 (5.5) | 1.1 (4.6) | 1.2 | 0.29–2.15 | 1.010 |
* Comparisons made between means of current and ideal time allocation using t-test; †only p values < 0.001 are considered to represent statistical significance.
Table 8.
Program director (PD) perception of resident current time allocation versus their ideals
| PD perception | Mean percent of time allocated to domain | 95% confidence interval | p value* | |||
|---|---|---|---|---|---|---|
| PD perception of current time allocation (SD) (n = 55) | PD perception of ideal time allocation (SD) (n = 55) | Mean difference | ||||
| Domains of resident responsibilities | ||||||
| 1 | Operating or performing procedures | 28.3 (12.9) | 32.6 (14.6) | −4.3 | −9.51 to 0.91 | 0.105 |
| 2 | Evaluating patients in a clinical setting | 21.1 (8.8) | 23.0 (12.3) | −1.8 | −5.89 to 2.20 | 0.369 |
| 3 | Caring for inpatients in the hospital | 14.9 (11.5) | 12.0 (8.3) | 2.9 | −0.87 to 6.69 | 0.130 |
| 4 | Evaluating patients in an acute care/emergency department setting | 8.9 (6.7) | 9.0 (5.2) | −0.1 | −2.36 to 2.18 | 0.937 |
| 5 | Independent studying or preparing for surgery/clinic/lectures | 8.3 (5.4) | 6.9 (6.2) | 1.3 | −0.85 to 3.53 | 0.227 |
| 6 | Attending formal lectures/courses | 4.9 (2.5) | 3.7 (2.4) | 1.2 | 0.28–2.12 | 0.011 |
| 7 | Answering patient phone calls/messages/questions/etc outside of clinical setting | 2.6 (2.9) | 1.4 (1.9) | 1.2 | 0.30–2.18 | 0.010 |
| 8 | Completing paperwork (prescriptions, disability, insurance forms, duty hours, etc | 3.6 (4.1) | 1.5 (1.9) | 2.1 | 0.88–3.30 | < 0.001† |
| 9 | Conducting clinical or basic science research | 4.8 (4.4) | 5.7 (5.8) | −1.0 | −2.93 to 0.97 | 0.321 |
| 10 | Deliberate practice of surgical skills outside of the operating room setting | 2.5 (3.5) | 4.0 (5.2) | −1.5 | −3.20 to 0.14 | 0.072 |
| 11 | Other | 0.2 (0.7) | 0.3 (1.1) | −0.1 | −0.39 to 0.29 | 0.772 |
| Resident roles in the operating room | ||||||
| 1 | Primary surgeon (making surgical decisions and performing the operation | 34.0 (29.0) | 43.3 (31.8) | −9.3 | −20.76 to 2.26 | 0.114 |
| 2 | First assistant | 45.1 (21.2) | 40.1 (21.2) | 5.0 | −2.97 to 13.05 | 0.215 |
| 3 | Second assistant | 18.7 (27.8) | 13.6 (25.1) | 5.1 | −4.92 to 15.10 | 0.316 |
| 4 | Observation only (not scrubbed in) | 2.2 (9.1) | 2.0 (6.4) | 0.2 | −2.77 to 3.19 | 0.889 |
* Comparisons made between means of current and ideal time allocation using t-test; †only p values < 0.001 are considered to represent statistical significance.
Discussion
The methods used to train current orthopaedic surgery residents are quite different from prior generations. With the introduction of duty-hour regulations, a changing economic climate, decreased educational contact time, emerging technologies such as simulation, and an ever expanding orthopaedic knowledge base, the evolution of orthopaedic surgical education is moving at a rapid pace [8, 12, 15]. Many programs are altering their curricula to meet these demands in an attempt to continue to graduate skilled orthopaedic surgeons from their programs. For current programs to continue to make progress in this regard, it is important that we understand how residents actually spend their time in training. Equally important, we should determine what would be the optimal or ideal way for residents to spend those 5 years of training to maximize the potential for them to progress from novice to competent to possibly even expert-level performers. To further this understanding, we collected responses to these questions from residents and PDs with an interest in optimizing resident education.
Limitations
Like with all survey studies, this report is limited by the potential recall and reporting bias of the respondents. It is also limited in that it was not conducted on a national level but instead included 11 programs belonging to the MOSSC and may not be universally generalizable to every program in the country. We intentionally chose to include only those programs involved in the MOSSC because this group has been assembled with the expressed intent of improving orthopaedic surgical education through outcomes-based research and interventions. We felt that the group was diverse enough in their program size, resident complement, academic affiliation, and training curricula to provide meaningful results. This survey was strengthened by its high response rate of 66%, which is two to four times higher than what has been observed in similar studies of orthopaedic residents and PDs that were distributed on a national level [8, 12]. We chose to ask residents to provide percentages of time rather than actual hours. This was done to reduce bias that could be generated if residents felt pressured to alter responses to maintain compliance with any duty-hour restrictions. To our knowledge, this study represents the first report of how orthopaedic residents actually spend their time in training, and it is the first attempt to elucidate an ideal time allocation model.
As one may expect, the way residents spend their time differs based on PGY with seniors spending more time in the OR and clinic and less time caring for inpatients and seeing patients in the emergency department. Seniors also spend a larger portion of their operative time acting as the primary surgeon compared with their more junior counterparts. Although little is known about how resident roles and responsibilities should evolve as learners progress through training, other work has demonstrated improved satisfaction and learning capacity among surgical residents when the ratio of service obligations to educational experiences is reduced [14]. Perhaps this is what is occurring as orthopaedic residents advance to higher levels of seniority.
Overall, PDs appear to have a good understanding of how residents perceive their time is spent during residency. The only discrepancy noted was that PDs overestimated the time residents spend serving as the primary surgeon when operating. This may be attributable in part to the fact that PDs have an obvious vested interested in the education of the residents rotating on their services, and they may tend to allow that learner to serve as a primary surgeon more often than what is permitted by other staff surgeons. Additionally, perceptions of what constitutes the role of primary surgeon may differ between residents and PDs. For instance, residents may not feel that they were the primary surgeon unless they do all of the decision-making and execution, whereas PDs may feel the resident is autonomous if they execute the majority of the case.
Compared with how they currently spend their time, residents feel that their training could be improved if they spent more time operating, serving as the primary surgeon, and deliberately practicing surgical skills outside of the OR. All three of these domains are similar in that they represent a desire of residents to improve their surgical abilities before graduation. It is also possible that residents want to spend more time in the OR and serving as a primary surgeon because they tend to enjoy that activity more than some of the other domains listed. A stronger case for the desire to improve surgical skills is supported by the notion that residents want to spend more time deliberately practicing their surgical skills outside of the OR using basic task trainers, advanced simulators, cadaveric skills laboratories, or other modalities. Both residents and PDs would like to see more time dedicated to this domain. This may be attributable in part to current resident duty-hour restrictions that reduce the time a learner can actually be in the OR [2, 8, 12, 15]. Use of simulation may prove a to be a valuable adjunct because emerging research in surgical simulation suggests that observable improvement can be expected through deliberate practice of these skills [3, 4].
Overall, only minor differences existed between the PDs’ perspectives on current and ideal time constructs. This may suggest that PDs are pleased with the way their current programs are structured, which could be the result of the fact that they have designed and modified their current programs to mirror that which they consider to be an ideal training regimen. Although much discrepancy exists between the resident and PDs’ opinions of an ideal time construct, there were some areas of notable concordance. For instance, both groups trended toward desiring more time spent in the OR, serving as primary surgeon, conducting research, and deliberately practicing surgical skills. Both residents and PDs also felt that paperwork and phone calls should be reduced to less than 1% to 2% of their time. Of all of the domains studied, these areas of agreement may represent appropriate targets for future optimization.
Conclusions
Residents and PDs seem to agree on how time is currently spent in residency training. Some differences of opinions continue to exist regarding how an ideal program should be structured; however, this work identifies a few potential targets for improvement that are agreed on by both residents and PDs. These areas include increasing OR time, finding opportunities for deliberate practice of surgical skills outside of the OR, and decreased clerical burden. This study may serve as a template for programs to continue to refine their educational models in an effort to achieve curricula that meet the desired goals of both learners and educators. Additionally, it is an initial step toward more objective identification of the optimal educational structure of an orthopaedic residency program. Future study into this topic is indicated.
Acknowledgments
We thank the residents and program directors of the MOSSC for their willingness to participate in this work.
Footnotes
Each author certifies that he or she, or a member of his or her immediate family, has no funding or commercial associations (eg, consultancies, stock ownership, equity interest, patent/licensing arrangements, etc) that might pose a conflict of interest in connection with the submitted article.
All ICMJE Conflict of Interest Forms for authors and Clinical Orthopaedics and Related Research ® editors and board members are on file with the publication and can be viewed on request.
Clinical Orthopaedics and Related Research ® neither advocates nor endorses the use of any treatment, drug, or device. Readers are encouraged to always seek additional information, including FDA-approval status, of any drug or device prior to clinical use.
Each author certifies that his or her institution approved the human protocol for this investigation, that all investigations were conducted in conformity with ethical principles of research, and that informed consent for participation in the study was obtained.
This work was performed at the Mayo Clinic, Rochester, MN, USA, and the University of Iowa Hospitals and Clinics, Iowa City, IA, USA.
References
- 1.ACGME. ACGME Milestones Project. Chicago, IL. Available at: www.acgme.org/acgmeweb/tabid/430/ProgramandInstitutionalAccreditation/NextAccreditationSystem/Milestones.aspxwww.acgme.org. Accessed January 27, 2015.
- 2.Baldwin K, Namdari S, Donegan D, Kamath AF, Mehta S. Early effects of resident work-hour restrictions on patient safety: a systematic review and plea for improved studies. J Bone Joint Surg Am. 2011;93:e5. doi: 10.2106/JBJS.J.00367. [DOI] [PubMed] [Google Scholar]
- 3.Cannon WD, Garrett WE, Jr, Hunter RE, Sweeney HJ, Eckhoff DG, Nicandri GT, Hutchinson MR, Johnson DD, Bisson LJ, Bedi A, Hill JA, Koh JL, Reinig KD. Improving residency training in arthroscopic knee surgery with use of a virtual-reality simulator. A randomized blinded study. J Bone Joint Surg Am. 2014;96:1798–1806. doi: 10.2106/JBJS.N.00058. [DOI] [PubMed] [Google Scholar]
- 4.Crochet P, Aggarwal R, Dubb SS, Ziprin P, Rajaretnam N, Grantcharov T, Ericsson KA, Darzi A. Deliberate practice on a virtual reality laparoscopic simulator enhances the quality of surgical technical skills. Ann Surg. 2011;253:1216–1222. doi: 10.1097/SLA.0b013e3182197016. [DOI] [PubMed] [Google Scholar]
- 5.Ericsson KA. Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad Med. 2004;79:S70–81. doi: 10.1097/00001888-200410001-00022. [DOI] [PubMed] [Google Scholar]
- 6.Ericsson KA. An expert-performance perspective of research on medical expertise: the study of clinical performance. Med Educ. 2007;41:1124–1130. doi: 10.1111/j.1365-2923.2007.02946.x. [DOI] [PubMed] [Google Scholar]
- 7.Ericsson KAK, Tesch-Romer C. The role of deliberate practive in the acquition of expert performance. Psych Rev. 1993;100:363–406. doi: 10.1037/0033-295X.100.3.363. [DOI] [Google Scholar]
- 8.Karam MD, Pedowitz RA, Natividad H, Murray J, Marsh JL. Current and future use of surgical skills training laboratories in orthopaedic resident education: a national survey. J Bone Joint Surg Am. 2013;95:95. doi: 10.2106/JBJS.L.00177. [DOI] [PubMed] [Google Scholar]
- 9.Kirkman MA. Deliberate practice, domain-specific expertise, and implications for surgical education in current climes. J Surg Educ. 2013;70:309–317. doi: 10.1016/j.jsurg.2012.11.011. [DOI] [PubMed] [Google Scholar]
- 10.Kulasegaram KM, Grierson LE, Norman GR. The roles of deliberate practice and innate ability in developing expertise: evidence and implications. Med Educ. 2013;47:979–989. doi: 10.1111/medu.12260. [DOI] [PubMed] [Google Scholar]
- 11.Marcus H, Vakharia V, Kirkman MA, Murphy M, Nandi D. Practice makes perfect? The role of simulation-based deliberate practice and script-based mental rehearsal in the acquisition and maintenance of operative neurosurgical skills. Neurosurgery. 2013;72(Suppl 1):124–130. doi: 10.1227/NEU.0b013e318270d010. [DOI] [PubMed] [Google Scholar]
- 12.Mir HR, Cannada LK, Murray JN, Black KP, Wolf JM. Orthopaedic resident and program director opinions of resident duty hours: a national survey. J Bone Joint Surg Am. 2011;93:e1421–1429. doi: 10.2106/JBJS.K.00700. [DOI] [PubMed] [Google Scholar]
- 13.Philibert I, Friedmann P, Williams WT. New requirements for resident duty hours. JAMA. 2002;288:1112–1114. doi: 10.1001/jama.288.9.1112. [DOI] [PubMed] [Google Scholar]
- 14.Reinke CE, Kelz RR, Pray L, Williams N, Bleier J, Murayama K, Morris JB. Trimming the fat: optimizing overall educational value by defining factors associated with overall educational value and service-to-education ratio. J Surg Educ. 2012;69(813–818):e811. doi: 10.1016/j.jsurg.2012.05.007. [DOI] [PubMed] [Google Scholar]
- 15.Robbins L, Bostrom M, Craig E, Sculco TP. Proposals for change in orthopaedic education: recommendations from an orthopaedic residency directors’ peer forum. J Bone Joint Surg Am. 2010;92:245–249. doi: 10.2106/JBJS.I.00210. [DOI] [PubMed] [Google Scholar]