A 15-year review of the Stanford Internal Medicine Residency Program: predictors of resident satisfaction and dissatisfaction

Abstract

Introduction

Satisfaction with training and with educational experiences represents important internal medicine (IM) programmatic goals. Graduates from IM residency programs are uniquely poised to provide insights into their educational and training experiences and to assess whether these experiences were satisfactory and relevant to their current employment.

Methods

We surveyed former IM residents from the training program held during the years 2000–2015 at the Department of Medicine, Stanford University. The first part of the survey reviewed the IM residency program and the second part sought identifying data regarding gender, race, ethnicity, work, relationships, and financial matters. The primary outcome was satisfaction with the residency experience.

Results

Of the 405 individuals who completed the Stanford IM residency program in the study period, we identified 384 (95%) former residents with a known email address. Two hundred and one (52%) former residents responded to the first part and 185 (48%) answered both the parts of the survey. The mean age of the respondents was 36.9 years; 44% were female and the mean time from IM residency was 6.1 (±4.3) years. Fifty-eight percent reported extreme satisfaction with their IM residency experience. Predictors associated with being less than extremely satisfied included insufficient outpatient experience, insufficient international experience, insufficient clinical research experience, and insufficient time spent with family and peers.

Conclusion

The residents expressed an overall high satisfaction rate with their IM training. The survey results provided insights for improving satisfaction with IM residency training that includes diversifying and broadening IM training experiences.

Introduction

Internal medicine (IM) residency programs seek to develop the knowledge, skills, and experiences among their trainees that contribute to autonomy, competence, and learning habits necessary for the growth of their graduates’ careers. To achieve these goals, IM residency programs create policies and training experiences while following rules established by the Accreditation Council for Graduate Medical Education (ACGME).¹ While resident passing rates on the Certification Examination of the American Board of Internal Medicine provide a measure of the cognitive medical knowledge attained by the residency graduates, these scores do not address other important aspects of training such as whether residents were prepared for their actual post-residency careers.^2,3 IM training influences practice behaviors and work-life balance; however, the ability of IM residency programs to prepare graduates for their subsequent careers is unclear.^4–8

We sought to evaluate the IM residency program conducted at the Department of Medicine, Stanford University, by surveying IM residents graduating during the past 15 years (2000–2015). The objective of the project was to learn how former IM residents viewed their training opportunities in relation to their present work and life experiences. We report the experiences and circumstances associated with satisfaction with the residency experience and suggest changes in the program that might improve the IM residency.

Methods

We developed a two-part survey to collect information about experiences during the IM residency program and about current work and life situations. The survey collected information using a Likert scale, and the scale was applied in a consistent manner to each survey question. Study data were collected and managed using REDCap electronic data capture tools hosted at the Stanford Center for Clinical Informatics. REDCap is a secure, web-based application designed to support data capture for research studies providing 1) an intuitive interface for validated data entry; 2) audit trials for tracking data manipulation and export procedures; 3) automated export procedures for seamless data downloads to common statistical packages; and 4) procedures for importing data from external sources.⁹ This project was reviewed by the Stanford University and the Veteran’s Administration (VA) Palo Alto Health Care System’s combined Institutional Review Board, and it was determined that the project did not meet the Federal definition of research or clinical investigation under DHHS 45 CFR 46.102(d) or clinical investigation (FDA 21 CFR 50.3©, 56.102(d)) and informed consent was not required.

Survey population

We surveyed former residents who completed IM training between 2000 and 2015. Residents were contacted by email, and three reminders were sent to complete the surveys.

Residency program overview

The Stanford IM Residency Program had a total of 121 IM residents in 2015. The program has experienced continued growth from a total of 89 IM residents in 2000 to 104 IM residents in 2008. The major training experiences include the inpatient service at Stanford University, VA Palo Alto Health Care System, and Santa Clara County hospitals, with a majority of the time spent at the first two sites (including the intensive care units [ICUs] at those two sites). IM residents typically spend 1–2 months of the year at the county hospital. All residents have a half-day continuity primary care clinic, and there is an additional ambulatory training on dedicated rotations, particularly concentrated in the second and third years of residency. In 2009, the outpatient experience was substantially expanded, increasing the ambulatory time in medical subspecialty clinics. Other notable changes during the survey period include the creation of a Global Health track and enhanced global health opportunities for IM residents in 2008 and a change in the IM Residency Program Director and Associate Program Directors in 2009. In 2011, the program implemented changes associated with the ACGME work-hour regulations. Since 2011, the majority of general medicine inpatient ward attendings have been career hospitalists, as compared to subspecialists or primary care generalists who comprised the majority of the general medicine inpatient ward attending group in the early part of the study period.

Primary outcome: satisfaction with residency

Satisfaction with the IM residency program was rated as extremely satisfied, somewhat satisfied, neutral, somewhat dissatisfied, and dissatisfied. Survey responders were classified as top box (extremely satisfied) or being less than extremely satisfied (the remaining four classifications) with their IM residency. Educational experiences were rated as “much less would have been better”; “a little less would have been better”; “adequate”; “somewhat more”; and “much more”; or “not sure.”

Statistical analyses

We divided the entire cohort into three 5-year groups. We compared those who did and did not state that they were extremely satisfied with IM residency experiences using 2-sample, unpaired t-tests for continuous variables and chi-square tests for categorical variables. To determine the relationship between extreme satisfaction with the overall residency experience and satisfaction with the individual aspects of the program, we conducted test for trend (nptrend, STATA, College Station, TX, USA). For multivariable analyses, we coded each Likert response to the individual aspects of the program as 0 (amount adequate), 1 (wanted a little more or less), or 2 (wanted a lot more or a lot less). We then used logistic regression with backward selection (required p<0.05 to remain in the model) to determine which of the individual aspects of residency were significantly associated with being extremely satisfied with the residency overall. All analyses were conducted using STATA 11.1.

Results

There were 405 former Stanford IM residents from 2000 to 2015. All but one had an email address on file with the Stanford IM residency program. Of the remaining 404 former residents, 123 (30%) emails generated either a nondelivery report/receipt or failed delivery status (“bounced”). Using Internet search engines, we were able to find accurate email addresses for 103 (84%) of these intended recipients. Thus, 384 former IM residents had an identified and valid email address and received the survey, and 201 (52%) completed it. The second part was only sent to the 201 initial respondents, and it was answered by 185 former residents (48% of the entire cohort; 93% of the initial respondents) (Table 1).

Table 1.

Characteristics of internal medicine resident

	Total cohort with an email address	Nonrespondents sent the first part of the survey	Respondents to the first and second parts to the survey	p-value
	n=384	n=183	n=185
Characteristics
Age, years (mean ± SD)	37.7±4.9	38±4.6	36.9±5.0	0.002*
Years from residency (mean ± SD)	6.7±4.2	7.4±4.1	6.1±4.3	0.002*
Female (%)			78/179 (44)
Race (%)
White			84/169 (50)
Asian			76/169 (45)
Black			2/169 (1)
Hispanic			5/169 (3)
Native American			2/169 (1)
Married (%)			130/177 (73)
Debt (%)
≤$50,000			92/183 (50)
>$50,000 and ≤$100,000			32/183 (17)
>$100,000 and ≤$200,000			34/183 (19)
>$200,000			25/183 (14)

Note:

^*Respondents compared with nonrespondents.

The respondents, compared to the nonrespondents, were significantly more likely to have recently completed their residency training (6.1 years vs 7.4 years, respectively) and were significantly younger than the nonrespondents (36.9 years vs 38.4 years, respectively) (Table 1, p=0.002 comparing responders with nonresponders). There was no significant difference in age or time from residency between respondents completing only the first part and those completing both parts of the survey (data not shown). Among the respondents, 78 (44%) were women, 84 (50%) were white, the mean age was 36.9 years (range 28–48), the mean time from completion of residency was 6.1 (±4.3) years, and 130 (71%) were married. Approximately half completed their residency with debt <$50,000 and 14% reported debt >$200,000. Ninety-seven percent of the graduates are full-time employed (89%) or part-time (8%). Cardiology was the most common subspecialty (16%) among the respondents, but the majority of respondents listed nonspecialist/other/unknown as “the fellowship that best reflects” their current employment (Table 2). Income was reported by 175 respondents and was <$150,000 for 65 (35%); 60 (34%) have an income between $151,000 and $250,000; 27 (15%) have an income between $251,000 and $350,000; 20 (11%) have an income between $351,000 and $500,000; and 6 (3%) have an income >$500,000. The majority of former IM residents live in California, and most of those in California live within 100 miles of Stanford University.

Table 2.

Professional activities

Practice area	N (%)
Cardiology	30/185(16)
Gastroenterology	18/185 (10)
Hematology/oncology	16/185 (9)
Infectious disease	16/185 (9)
Rheumatology	11/185 (6)
Pulmonary/critical care	7/185 (4)
Nephrology	6/185 (3)
Endocrinology	4/185 (2)
Nonsubspecialist/other/unknown	77/185 (42)

One hundred and seventeen (58%) of the respondents were extremely satisfied with their experience with the Stanford IM residency program; there was no significant difference in program satisfaction by time since completing residency (Figure 1). The majority indicated that their clinical encounters were “about right” with regard to their educational experiences for outpatient, inpatient, and ICU training (Figure 2A). More county public hospital training was desired by some respondents; 40% of the residents would have wanted “more or much more” training in the county setting compared to 5% requesting “more or much more” training at the University Hospital or the VA Hospital. These former IM residents felt that they spent about the right amount of time with their peers but would have wanted more or much more time with family members (Figure 2B). There was no difference in satisfaction among former residents based on teaching by hospitalists or nonhospitalists (Table 3). More or much more “bedside teaching” was desired by 30% of the respondents, similar to the request for greater international health training. Basic research exposure was “about right”; however, 53% of residents wished they had more or much more exposure to clinical research (Table 3). Based on their current work experience, respondents suggested that they would have preferred more training in quality improvement, information technology, business management, and more experience with administrative leadership (Table 3).

Figure 1.

Satisfaction with internal medicine residency is displayed for different years of residency completion.

Notes: The majority of graduates were extremely satisfied. The differences between years of completion groups were not statistically significant (analysis of variance 0.6178; p=0.07 uncorrected for multiple comparisons).

Figure 2.

(A) Views of former residents regarding whether different types of internal medicine experiences should be changed are displayed. (B) Responses of former residents when asked for suggestions regarding satisfaction with the amount of time spent with peers and family.

Table 3.

Adjusted analysis of IM resident educational experiences and the level of satisfaction for the experience

View on adequacy of time spent	Highly satisfied (%)					Less than highly satisfied (%)					p-value	Adjusted p-value
	Much less	Less	Neutral	More	Much more	Much less	Less	Neutral	More	Much more
Outpatient	0 (0)	2 (2)	66 (58)	34 (30)	12 (11)	0 (0)	2 (2)	30 (36)	36 (43)	16 (19)	0.002	0.007
Inpatient	0 (0)	20 (17)	93 (80)	3 (3)	0 (0)	2 (2)	22 (26)	59 (70)	1 (1)	0 (0)	0.14	NS
Intensive care unit	0 (0)	8 (7)	91 (79)	13 (11)	3 (3)	0 (0)	14 (17)	58 (69)	9 (11)	2 (2)	0.17	NS
University Hospital	0 (0)	6 (5)	105 (90)	6 (5)	0 (0)	1 (1)	8 (10)	74 (88)	1 (1)	0 (0)	0.82	NS
Veterans Hospital	2 (2)	8 (7)	102 (88)	4 (3)	0 (0)	0 (0)	10 (12)	70 (84)	2 (2)	1 (1)	0.45	NS
County hospital	2 (2)	3 (3)	65 (57)	37 (32)	8 (7)	0 (0)	5 (6)	41 (49)	28 (33)	10 (12)	0.23	NS
Skills
Administration	1 (1)	2 (2)	59 (51)	39 (34)	6 (5)	0 (0)	4 (5)	29 (35)	32 (38)	13 (15)	0.008	NS
Information technology	2 (2)	3 (3)	58 (51)	35 (31)	12 (11)	2 (2)	3 (4)	33 (39)	35 (31)	12 (11)	0.11	NS
Quality improvement	2 (2)	14 (12)	56 (48)	32 (28)	7 (6)	0 (0)	9 (11)	41 (49)	23 (27)	8 (10)	0.74	0.026
LEAN–process improvement	4 (3)	13 (11)	33 (28)	32 (33)	7 (6)	3 (4)	8 (10)	31 (37)	21 (25)	7 (8)	0.48	NS
Business of medicine	0 (0)	2 (2)	23 (20)	53 (46)	34 (29)	2 (2)	4 (5)	12 (14)	29 (35)	37 (44)	0.073	NS
Bedside teaching	0 (0)	3 (3)	84 (72)	24 (21)	4 (3)	0 (0)	4 (5)	44 (54)	27 (33)	7 (9)	0.005	NS
Interaction with hospitalists	1 (1)	8 (7)	88 (76)	14 (12)	4 (3)	0 (0)	10 (12)	52 (62)	18 (21)	3 (4)	0.038	NS
Interaction with nonhospitalists	1 (1)	6 (5)	84 (72)	21 (18)	3 (3)	2 (2)	10 (12)	49 (58)	20 (24)	2 (2)	0.073	NS
International experience	2 (2)	2 (2)	59 (51)	28 (23)	2 (2)	1 (1)	3 (4)	32 (38)	29 (35)	9 (11)	0.007	0.01
Basic research	1 (1)	9 (8)	83 (72)	16 (14)	0 (0)	4 (5)	10 (12)	53 (64)	5 (6)	6 (7)	0.33	NS
Clinical research	0 (0)	2 (2)	56 (48)	46 (40)	10 (9)	1 (1)	3 (4)	29 (35)	29 (35)	21 (25)	0.010	0.033
Family time	0	0	88 (77)	27 (23)	0	0	1	31 (37)	39 (46)	13 (15)	<0.001	<0.001
Peer time	0	1 (1)	55 (34)	7 (6)	0	0	1 (1)	55 (34)	17 (21)	9 (11)	<0.001	0.033

Note: Adjusted analysis: coded much less or much more as 2, less or more as 1, and neutral as 0.

Abbreviation: NS, not significant.

We performed a multivariable analysis for the educational experiences that were most associated with being less than extremely satisfied (Table 3). Lesser levels of satisfaction were associated with less time spent with family (p<0.0001) and peers (p=0.03) and less time devoted to the following: outpatient care (p=0.006), international experience (p=0.009), training in quality improvement (p=0.03), and clinical research (p=0.03). Neither age nor years since residency were significantly associated with satisfaction (data not shown).

Discussion

Overall, 58% of the IM residents trained at Stanford during the past 15 years who responded to our survey reported being extremely satisfied with their educational experiences. There was variation among respondents reporting satisfaction with their IM experience on a year-to-year basis; however, there was no statistical difference among the 5-year groupings (p=0.07). Of note, the middle period corresponded to pre-ACGME work rule implementation, changes in leadership within the IM residency program, lack of a structured international educational experience for residents, and the development of a nascent hospitalist program. Unfortunately, our sample was not large enough to demonstrate important effects of these changes on satisfaction.

Bedside teaching and exposure to teaching by hospitalists’ and nonhospitalists were “about right,” suggesting that the educational activities by both groups of attending physicians were equally valued by the IM residents. This is a different finding from other studies and suggests that hospitalists and nonhospitalists provide complementary educational value for IM training.^10–12 A desire for greater exposure to training in clinical research, quality improvement, medical administration, information technologies, and business management was reported. These areas now receive greater educational attention, and some of these experiences include dedicated rotations, core curriculum conferences, and integration into clinical rotations. Given work-hour/training-length restrictions by ACGME, any increase in new educational activities may decrease training in the existing IM settings.^2,7,13–21 However, providing new and innovative educational opportunities to empower resident choice may increase resident training satisfaction and educational value.

Former IM residents were satisfied with their inpatient and intensive care experiences and desired more outpatient training. We suspect this reflects that the outpatient setting is a more common setting of care among the graduates of the IM residency, while the inpatient wards were the most common setting of their IM training. In traditional IM residency programs, outpatient experiences tend to have less dedicated time for education and less structured team learning, and we speculate that there may be less frequent feedback compared with inpatient training. This difference may limit certain types of education experiences for the resident and limit the development of resident autonomy.^22–35 IM residency programs may emphasize inpatient subspecialty consultations to broaden training experiences expecting that the subspecialty fellowship represents the key opportunity for outpatient training. We speculate that enhanced experiences in outpatient clinics, including subspecialist clinics, may help IM residents prepare for increasingly common patient care situations and interactions among practicing physicians.

The response rate (52% to the first part of the survey) was higher than expected.^{17,18,36–38} Responders were more likely to be recent graduates than the nonresponders. This may indicate that the more recently graduated IM residents had deeper connections to the IM residency program or perhaps had relationships to current residents and thus a desire to help shape and improve their experiences. It is also possible that the survey provided former IM residents with the opportunity to reflect on their training, and self-reflection may have been the motivation for completing the survey. Survey completion may be a marker for the willingness to engage with the training program and provides the program leaders with important perspectives on the training delivered.

The limitations of this project include recall bias, response bias, and the survey’s lack of specificity for respondents to identify their careers as hospitalists or primary care internists. Recall bias may influence our results if time clouds the memories of the former IM resident by preferentially recalling either the best experiences or the worst experiences; however, this survey asked the residents to rate their IM resident experiences as it influenced their present work situation and so recall bias may not have been a prominent limitation of this study.

Response bias could explain our results since respondents could be different than nonrespondents. Nonrespondents could have a different level of satisfaction with their training experiences than the respondents. We would have expected higher response rates and higher satisfaction from recent IM graduates familiar with the latest program changes aimed at diversifying the experiences and improving the satisfaction with the residency; however, this was not observed. Neither age at graduation, training era, or years from graduation was significant factor for programmatic satisfaction. Thus, response bias was unlikely to skew our results. We hope that providing these results to all our former residents may increase their response to subsequent surveys.

We provided a complete list of approved fellowships for the former residents to choose from, and cardiology was the most common subspecialty. We did not provide a choice for primary care or hospitalist (neither requires a fellowship). We assumed that most respondents not selecting a fellowship would have chosen hospitalist or primary care jobs to describe their present work situation, but this was an error in our survey construction. Future surveys will be crafted to include non-fellowship-based careers as well as subspecialty careers.

Although this study was imperfect, the results provide some sensible internal validity: extreme satisfaction is reduced with less time spent with family and peers. External validity is unknown, and the investigators encourage other programs to ascertain the opinions of graduates of their training programs. We anticipate that the results from our survey regarding time spent with family and peers would be similar to the results from other similar IM residency programs, although these data are infrequently collected and reported.

It is not surprising that the less time residents spent with family and peers, the less satisfied they were with their educational experiences. This association suggests the possibility and supports our speculation that at its core, an IM residency program should encourage relationships and time spent with peers and family. Mentoring programs that bring small groups of residents together, informal and formal social events, and attention to the work-life balance of IM residents are likely critical for resident satisfaction with the IM residency program.

There was a considerable debt among IM residents, with 50% acknowledging >$50,000 of debt when their IM training ended. We speculate that debt reduction linked to service during IM training will enhance their level of satisfaction with their IM residency programs could be a valuable tool for IM residency programs.^39,40

IM residency program leaders implement changes to meet the demands and expectations of IM residents and to prepare the trainees for their future careers. Recently, the Stanford IM Residency Program has made a substantial commitment to change by introducing a formal mentoring process and multiple pathways of distinction (allowing residents to share an interest with faculty and other residents to focus on the following areas: basic research, innovation and biodesign, primary care, clinical and bedside teaching, clinical research, quality improvement, and care of underserved populations). We have also continued to expand our Global Health program that now includes a fully supported 6-week overseas educational experience through a partnership with Johnson & Johnson. Like many IM residency programs, we implement changes understanding the risks associated with residents prematurely selecting career paths. Not all changes may succeed, and we continue to evaluate our program. In that regard, we believe that the results from this survey provide the opportunity for leaders of other IM residency programs to reflect on their training programs and make changes that their future trainees would value.