Effectiveness of a Leadership Development Course for Chief Residents: A Longitudinal Evaluation

Abstract

Background

Nonclinical skills (eg, self-regulation, team leadership, conflict resolution) are essential for success as a chief resident (CR). The literature on programs teaching these skills reports few if any effectiveness outcomes.

Objective

We reported the outcomes of a leadership course for CRs using participants' self-reported outcomes and assessments from their program directors (PDs).

Methods

A 2-day curriculum focused on emotional intelligence competencies, including self-awareness, self-management, social awareness, and relationship management. We used a logic model to align 2017–2018 curriculum with targeted outcomes. Questionnaires before and after the course assessed short-term and intermediate outcomes for the participants and PD interviews evaluated observed changes in CRs' performance attributable to the course.

Results

A total of 74 residents participated in the course, and 65% and 59% responded to the post-course and follow-up questionnaires, respectively. Over 95% of respondents indicated developing leadership knowledge and skills and connecting with new CRs in the post-course questionnaire. During follow-up, CRs reported applying concepts learned during chief residency, using tools to address conflict, engaging in quality and patient safety projects, and continuing to interact with other participants. The relationships between reported outcomes and participants' gender/prior leadership training were not significant (P > .05), with small to medium effect sizes (0.01–0.32). All 14 PDs offered positive appraisal of the CRs, but we could not specifically attribute this growth to the course.

Conclusions

Participation in this CR leadership development course was associated with enhancement and application of leadership competencies in immediate and intermediate time frames.

What was known and gap

Being a successful chief resident (CR) requires several nonclinical skills, but the literature does not report the effectiveness of programs that teach these skills to residents.

What is new

A leadership course for CRs using participants' self-reported outcomes and assessments from their program directors.

Limitations

The study examined self-reported outcomes, and it was not possible to attribute skill growth among CRs directly to the course.

Bottom line

Participation in the course was associated with enhancement and application of leadership competencies, in immediate and intermediate time frames.

Introduction

Being a chief resident is a role that requires management and leadership skills (eg, communicating, aligning groups, advocating, and implementing change). Recommended curricula of health systems science¹ and of CanMEDS² explicitly include leadership as a key competency for trainees. Yet, chief residents (CRs) frequently report low familiarity with these skills.^3–5 There is consensus regarding a need for leadership development for physicians and that such training should begin early in training.^6–8 For example, Blumenthal et al and others have highlighted residency as an opportunity for developing leadership^3,4,9–11; they recommend nurturing non-clinical skills such as self-reflection, self-awareness, and self-regulation; team leadership; change management; negotiation; and maintaining professional networks. Because residents provide a significant portion of the care in large teaching hospitals, those who undertake leadership responsibilities can have a significant impact on patient care.

In response to this need to develop leadership skills among CRs, various programs have been established in single specialties, multiple specialties, single institutions, and national programs.^12–21 Still, published evidence demonstrating the effectiveness of such programs remains sparse, with studies focusing mostly on immediate program outcomes (ie, improvement in participants' knowledge, self-awareness and self-confidence, and anticipated behavior changes^16,21–23) and self-reported outcomes. Several systematic reviews of physician leadership development programs have shown substantial evidence gaps, especially regarding long-term measures and high-impact outcomes.^24–26

The current study was undertaken to address these gaps. Specifically, we measured immediate and intermediate outcomes associated with a longstanding leadership workshop³ offered to all CRs at the Cleveland Clinic. Outcomes included participants' self-reports as well as assessments from the CRs' program directors (PDs) regarding workshop participant performance and application of these concepts during the subsequent CR year.

Methods

Site and Setting

Cleveland Clinic is a fully integrated, multispecialty academic medical center where 985 residents and fellows trained in Cleveland in 75 Accreditation Council for Graduate Medical Education (ACGME) accredited programs and 80 additional programs in the 2017–2018 academic year. At Cleveland Clinic Florida (CCF), 100 residents and fellows trained in 11 ACGME-accredited programs.

Since 2008, a 2-day Chief Residents Leadership Workshop has been offered annually to all CRs in the initial weeks of their chief residency year.³

Leadership Curriculum

The curriculum was based on longstanding leadership development programs at CCF (eg, Cleveland Clinic Learning Academy, Leading in Health Care, and Samson Global Leadership Academy).^3,27,28 These and the current curriculum are grounded in the emotional intelligence (EI) framework of Goleman et al.²⁹ Goals and topics addressed in the course (Table 1) correspond to 4 areas of EI competencies: self-awareness, self-management, social awareness, and relationship management. The 12 course sessions were taught by CCF faculty and selected extramural invited experts.

Table 1.

Features of the Chief Resident Leadership Course

Features	Details
Participants	Chief residents (CRs) from Cleveland Clinic (Ohio and Florida) and other hospitals in Northern Ohio who represent different genders and specialties
Faculty	Multidisciplinary faculty consisting of 6 Cleveland Clinic senior professional staff and 2 professors from other higher education institutions
Curricular goals	Nurture in participating CRs
	Self-knowledge of leadership strengths and weaknesses
	Appropriate use of different leadership styles
	Teamwork and team-building skills
	Understanding of the role of organizational culture and values in building effective health care organizations
	Knowledge of programs to promote quality and patient safety
	Knowledge of theories to change organizations and tools to measure that change
	Ability to resolve conflicts within an organization
	Understanding of the importance of cultural competence in a health care organization
	Knowledge of programs and resources to support physicians' well-being
Curriculum topics	The characteristics of effective leaders in health care
	Emotional intelligence as a paradigm for learning leadership skills
	Organizational culture
	Health care quality and patient safety
	Tools for effective communication for physician leaders
	Definition and tools to evaluate cultural competence in an organization
	Basic theories on how to change and improve organizations
	Basic tools to manage conflict
	The characteristics of effective health care teams
	Resources and programs to support physician well-being
Teaching methods	Presentations on curricular topics by faculty
	Facilitated, interactive, in-class group activities
Participant engagement	Individual tasks during course: completion of the Thomas–Kilmann Conflict Mode Instrument and individual pre-work for team-building exercise
	Participation in group tasks and discussions during course
	Pre- and post-course questionnaires
Duration	2 days: 8:00 am–5:30 pm on day 1; 8:00 am–4:15 pm on day 2
	12 individual sessions with scheduled breaks
	Session duration ranged from 45 minutes to 2.25 hours

Course Participants

Current CRs starting their chief residency tenure were nominated to attend by their PDs. They represented all specialties at CCF campuses in Cleveland, Akron, and Florida, and other health centers in Northern Ohio. The course, offered in early August, was announced in various venues, including the CCF GME website,³⁰ program leadership and council meetings, and to ACGME designated institutional officials at other hospitals in Northern Ohio who were invited to nominate CRs.

Course Evaluation

We applied a logic model³¹ to track and align course activities with resulting outcomes (Table 2). Components of this logic model include resources devoted to the course, activities targeted to attain desired outcomes, outputs of course participation, and outcomes from CR course participation. The following 2 questions were investigated in the 2017–2018 academic year course evaluation: What are the immediate (last day of course) and intermediate outcomes (9 months after course completion) for the CRs who participate in the course? What suggestions for improvement, if any, do stakeholders have about the course?

Table 2.

Logic Model for Chief Resident Leadership Course Evaluation

Resources/Inputs	Activities	Outputs	Outcomes
• Grant funding o Scholarship stipends to external chief residents for travel to workshop o Speakers' honoraria o Food for 2-day event o Photographer o Supplies o Evaluation and dissemination • Cleveland Clinic's support o Course directors (100% compensation for time devoted to program) o Program staff salary and fringe benefits o Audio/visual o Administrative costs • Faculty time and expertise • Course curriculum • Chief resident off-service time	Recruitment and enrollment • Course director's administrative efforts • Course announced on internal and external venues • Program directors selected chief residents to attend course • Disbursement of scholarships to eligible chief residents • Communication regarding enrollment and participation Design and implementation of course • Faculty invitations • Designed and/or modified course curriculum • Development of didactic and interactive teaching methods o Chief residents completed assigned pre-work o Chief residents engaged in group activities during course Assessment of chief residents • Chief resident self-assessment of learning and application • Program director appraisals	• 74 chief residents o 64% male o 64 chief residents from Cleveland Clinic o 10 chief residents from 4 other health systems across Northern Ohio • Chief residents represented 28 specialties • 2-day course	Immediate • Improved knowledge and skills related to leadership and management in health care (self-efficacy) • Expanded network of chief resident peer group support Intermediate (follow-up study) • Increased application of leadership and management concepts on the job • Development and use of tools to resolve conflict • Initiation and implementation of quality improvement projects • Favorable appraisal of chief residents' performance and potential by their residency program directors • Continued interaction with chief resident peer group

Data Collection

Data collection for this mixed methods evaluation included questionnaires and interviews. All data collection instruments were piloted and modified based on feedback from medical students in a similar leadership course (pre- and post-course questionnaires), prior CR course attendees (CR follow-up questionnaire), and former PDs (PD interview guide). These documents are provided as online supplemental material.

We used a pre-course questionnaire to assess participants' baseline familiarity with key concepts covered in the course. To investigate short-term or immediate outcomes (Table 2), we administered questionnaires with closed- and open-ended questions to participants before and immediately after the course. Intermediate outcomes were assessed using an online questionnaire administered on REDCap, a web-based platform for developing and managing surveys (Vanderbilt University, Nashville, TN). Two authors (S.M. and C.F.F.) also interviewed 14 PDs purposefully selected³² to represent core programs (anesthesiology, family medicine, general surgery, internal medicine, neurological surgery, pediatrics, obstetrics and gynecology, ophthalmology, orthopedic surgery, pathology, pediatrics, psychiatry, radiology, and urology) that had the largest number of residents and who had sent multiple CR course participants. Three of these PDs were from other hospitals in Northern Ohio. Interviews were one-on-one, semistructured, and held either in PDs' offices or by telephone. Interview duration was approximately 10 to 30 minutes. Interview questions were developed from the course content and outcomes and assessed whether PDs observed any changes in CRs' behavior, attitude, or performance attributable to the course, as well as their overall evaluation of the course (interview guide provided as online supplemental material).

This continuous quality improvement project was deemed exempt by the Cleveland Clinic Institutional Review Board.

Data Analysis

Responses to the Likert scale questions on the pre- and post-course and 9-month follow-up questionnaires were entered into SPSS 24 (IBM Corp, Armonk, NY) to compute descriptive statistics of responses and summarize participants' demographic characteristics, including gender, specialty, institutional affiliations, and prior leadership training. CR responses to open-ended questions on the abovementioned questionnaires were analyzed using Dedoose (University of California, Los Angeles, CA), a web-based software for analyzing qualitative and mixed methods research, to uncover emerging themes related to the perceived immediate and intermediate course outcomes. Fisher's exact 2-tailed tests were used to assess differences in CRs' reported immediate and intermediate outcomes across levels of gender and prior leadership training. The a priori level of statistical significance was P < .05. Cramér's V statistic served as a measure of the effect size (ES). This statistic has a lower limit of 0 and an upper limit of 1. We used Cohen's criteria³³ to classify ES magnitude (0.10 = small, 0.30 = medium, and 0.50 = large). Data from the PD interviews were transcribed verbatim and coded using Dedoose. Two authors (S.M. and C.F.F.) performed a qualitative content analysis of PD interview transcripts independently and then discussed to reach consensus.³⁴

Results

Seventy-four new CRs (64% [47 of 74] male) representing 28 specialties and 5 health systems in Ohio and Florida attended the course. Participants included 64 CRs from Cleveland Clinic (Ohio and Florida) and 10 CRs from other medical centers across Ohio.

Achievement of Immediate Outcomes

Pre-Course Questionnaire:

All participants were queried regarding their perceived familiarity with different leadership competencies, including EI, cultural competence, organizational change, conflict resolution, teamwork, and team building. Most indicated levels of familiarity that were below moderate, which corresponded to a mean score of 4 points on a 5-point scale.

In open-ended responses, participants reported a number of motivations for taking this course. Among the top reasons were to (1) develop or improve leadership skills, (2) better serve as CRs, and (3) comply with the recommendation of their program leadership. Some CRs indicated being motivated by their future leadership career aspirations, not having received leadership training before, and personal interest in course content. Participants indicated various reasons for seeking future leadership opportunities in health care organizations, ranging from a desire to pursue careers in academic medicine, improve and innovate health care, or assume specific leadership roles, such as department chairs, residency PDs, and chief of staff.

Post-Course Questionnaire:

Forty-seven of 74 CRs (57% [27 of 47] male) responded to the post-course evaluation (64% response rate), most of whom (57%, 27 of 47) had not had prior leadership training. Over 95% (45 of 47) of respondents indicated they developed leadership knowledge and skills represented by the first 10 curricular goals, related learning objectives of the course, and connected with CR peers they did not know before (Table 3). All 47 respondents recommended the course to future CRs. Participants' responses to open-ended questions supported their quantitative ratings (Table 4). For example, respondents reported increased awareness of the characteristics of effective teams, change management strategies, and conflict styles and approaches to conflict resolution. No significant relationships were found between reported immediate outcomes and participants' gender or prior leadership training (P > .05). Effect sizes ranged from 0.01 to 0.32, suggesting weak to moderate associations between immediate outcomes and CRs' gender and prior leadership training.

Table 3.

Immediate Outcomes Reported by Chief Residents in Post-Course Questionnaire

After Participating in the Chief Residents Leadership Workshop:	na	Agree and Strongly Agree (%)
1. Self-knowledge of leadership strengths and weaknesses
I understand the basic principles of emotional intelligence	47	100
I recognize applications of emotional intelligence principles to health care leadership	47	100
I am more aware of my leadership strengths	47	98
I am more aware of my leadership areas for improvement	47	98
I learned strategies for communicating health care matters effectively with others (for example, colleagues, patients, and public)	45	96
2. Appropriate use of different leadership styles
I am able to discuss characteristics of effective leaders in health care	47	100
3. Teamwork and team-building skills
I can discern the characteristics of effective teams	47	100
4. Understanding of the role of organizational culture and values in building effective health care organizations
I understand how an organization's culture informs its strategy	47	100
I appreciate the role of health care leaders in influencing organizational culture	47	100
5. Knowledge of programs to promote quality and patient safety
I am able to describe what quality in health care means	47	98
I recognize the link between patient safety and health care quality	47	98
I gained knowledge of programs to promote quality and patient safety	47	96
6. Knowledge of theories to change organizations and tools to measure that change
I learned about theories/models of organizational change	47	100
I was provided with tools to measure organizational change	47	96
7. Ability to resolve conflicts within an organization
I am able to recognize conflicts better	47	100
I am better equipped to address conflicts at work	47	98
8. Understanding of the importance of cultural competence in a health care organization
I recognize the challenges in building cultural competence into an organization	47	100
9. Knowledge of programs and resources to support physician well-being
I learned about resources to support physician well-being	47	96
10. Other: building a successful career in medicine (wrap-up)
I am more mindful of factors that can impact my success in medicine	45	100
I am familiar with the essential steps toward building a successful career in medicine	45	100
11. Connect/network with other chief residents
I was able to connect with other chief residents whom I did not know before	45	96

^aSome of the 47 post-course questionnaire respondents did not answer all the questions, thus the “n” values for selected questions are less than 47.

Table 4.

Immediate Outcomes Reported by Chief Residents (CRs) in Open-Ended Responses

Course Goals: Nurture in CRs	Illustrative Quotes From Participants' Open-Ended Responses
Self-knowledge of leadership strengths and weaknesses	“Self-awareness of leadership style and ways to branch out.”
Appropriate use of different leadership styles	“Good leaders project caring and trustworthiness.” “Environment creation is the role of a leader.”
Teamwork and teambuilding skills	“Ways to effectively manage teams.” “Traits of successful teams.”
Understanding of the role of organizational culture and values in building effective health care organizations	“Understanding your place and how your personal culture help you navigate organizational culture.”
Knowledge of programs to promote quality and patient safety	“The aspect of quality and safety improving outcomes and value.”
Knowledge of theories to change organizations and tools to measure that change	“Implementing change strategies.”
Ability to resolve conflicts within an organization	“Learned to better deal with conflicts and different styles of doing so.” “I learned more tools for conflict resolution and how to approach communication in complicated scenarios.”
Understanding of the importance of cultural competence in a health care organization	“Health equity and improving access to care for our diverse patients and residents.”
Other: building a successful career in medicine (wrap-up)	“Being able to better rely on my specific leadership styles will significantly aid in my career as a chief.”
Connect/network with other CRs	“Meeting other CRs in different specialties and hearing their thoughts on common issues.” “I enjoyed meeting the other CRs within the enterprise.”

Achievement of Intermediate Outcomes

Follow-Up Questionnaire:

Forty-four of 74 CRs (61% [27 of 44] male) responded to the 9-month follow-up questionnaire (59% response rate). Twenty-three of 44 respondents (52%) reported prior leadership training; 4 of these CRs were from institutions other than CCF. Ninety-five percent (42 of 44) of the respondents indicated they had applied concepts learned in the course during their chief residency, and 82% (36 of 44) used tools learned in the course to address conflicts in their programs (Table 5). The most widely applied concepts included EI, conflict resolution, and communication, followed by leadership, teamwork, and team building. Forty-one percent (18 of 44) engaged in various quality and patient safety and other improvement projects (Table 5). Fifty-two percent (23 of 44) reported continued interactions with other participants through various media following the course (eg, discussions by e-mail, connecting at local conferences, and collaborating on didactics and joint patient care). Those who did not report staying in contact with other CRs cited lack of time, opportunity, or distance. CRs' gender and prior leadership training were not significantly associated with the above intermediate outcomes (P > .05) and the magnitude of associations were small (ES ≤ 0.20).

Table 5.

Intermediate Outcomes Reported by Chief Residents in Follow-Up Questionnaire (9 Months After Course)

Participants' Use/Application of Learning From Course During Chief Residency	No. (%) of “Yes” Responses (N = 44)	Illustrative Quotes From Participants' Open-Ended Responses
Applied concepts learned in the course during their chief residency	42 (95)	“I feel the emotional intelligence portion of the course was very helpful in understanding multiple points of view and using personal and others' emotions and reactions in a constructive way rather than knee-jerk reactions which may be counter-productive.” “Approaching conflict resolution by first trying to engage and examine perspective of all stakeholders and then trying to develop potential solutions in collaboration with each stakeholder.”
Used tools received from the course to address conflicts in their programs	36 (82)	“Better handling of inter-resident and resident-faculty conflicts has been achieved using some of the communication skills and conflict management strategies reviewed.”
Interacted with chief residents from other programs whom they met during the course	23 (52)	“I was able to keep in touch with a few of the medicine residents and we shared ideas through email on how to handle some issues that arose within our respective programs.”
Initiated any change or improvement projects in their programs that they attribute to participation in the course	18 (41)	“Created a QI curriculum that involves learning the tools of QI through patient examples. We went through root cause analysis, 5 whys, and PDSA. As a result, the culture toward safety has changed as well.”

Abbreviations: QI, quality improvement; PDSA, plan-do-study-act.

Program Director Interviews:

All 14 PDs offered positive appraisal of their CRs who had attended the course and expressed intent to continue nominating CRs for future offerings. Reasons offered for continued nominations of CRs ranged from exposing their chiefs to foundational leadership knowledge and skills to enabling and promoting a community of leaders in order to support one another and share experiences. While the PDs reported growth in the CRs over the course of their chief residency, attribution of this growth to the course was not possible, partly because some PDs were insufficiently familiar with the course curriculum. Others suggested that it would have been helpful if they had been alerted beforehand to assess CR performance in specific domains addressed by the course. Several PDs had discussed the impact of the course with their CRs before the study interviews.

Follow-up feedback from the PDs and CRs was favorable. Additional topics suggested by PDs who knew the course curriculum well included the importance of role modeling to juniors, giving feedback, leveraging one's own strengths, and practice management. Suggestions from CRs centered mainly on promoting more interactivity and including experiential learning activities like case-based scenarios, small group sessions, specific scenarios to help apply concepts taught in the workshop, and discussion of concrete examples whenever possible to illustrate a topic. Some suggested more facilitation of participant introductions and assigning seating to encourage interactions among newly acquainted attendees. To refresh the curriculum throughout the year, some PDs and CRs suggested periodic check-ins with the participants or webinars.

Course Expenses and Feasibility:

Course expenses mainly reflected the cost of travel and lodging for those CRs attending from long distances away, estimated at approximately $1,200 per person (supported by a grant), and an honorarium for one outside faculty member. Costs related to catering for 2 days were modest (∼$30 per attendee). In a full salary model like the Cleveland Clinic, costs related to course director (J.K.S. and C.F.F.) contribution of 2 full days and CRs' time were covered. Administration of the course was supported by a program manager for whom course organization was estimated to require 30 hours (∼0.02 full-time equivalent).

Discussion

In this mixed methods study investigating the outcomes of a leadership course for CRs from different specialties and health centers across Northern Ohio and Florida, baseline familiarity with leadership competencies was generally low and the self-reported impact of the course on participants' knowledge of leadership competencies was high. During 9-month follow-up interviews, PDs reported growth in their CRs. In line with feedback from the study participants, additional course time will be reserved in the future for CRs to discuss application of course content to their work and solve case-based scenarios in small groups. Course goals and objectives will be shared with PDs to help with assessment of CRs' performance and selection of CRs for the course.

This study extends available findings regarding leadership development for CRs in several ways. In their meta-analysis of leadership training in graduate medical education (GME), Sadowski et al³⁵ pointed out that “leadership curricula are heterogeneous and limited in effectiveness.” Furthermore, most available studies of leadership training programs in GME have assessed only short-term effects, most frequently in post-curriculum or pre-post surveys, and rarely included longitudinal follow-up with participants or measured higher-level outcomes at Kirkpatrick's level 3 (transfer of learning to workplace attributed to educational program) or level 4 (organizational/performance changes directly attributed to educational program).³⁶ In this study, CRs provided specific examples (9 months post-course completion) where they applied leadership tools to alleviate interpersonal conflicts encountered during chief residency (level 3) and implemented quality improvement projects based on course concepts (level 4). Thus, by obtaining both immediate and intermediate term outcomes, the current study addresses a gap in the available literature regarding physician leadership development.^24–26 Although attribution to the course was largely uncertain, the current study distinctively addressed impacts of the course on participants' leadership behaviors by surveying their PDs, and is unique in including CRs from different specialties and health systems in a single course.³⁷

Several limitations of the study warrant comment. Bias toward course impact was introduced because CRs are almost uniformly selected for their leadership potential.^38,39 Such selection bias and the lack of a control group precludes attributing outcomes to course participation. As with most other studies, this study included self-reported outcomes,⁴⁰ thereby falling short of more rigorous measures of leadership competency acquisition that might be provided by serial multi-rater feedback on EI or leadership competence. Finally, while CRs from multiple institutions participated, small numbers preclude assessing whether the impact of the course is generalizable beyond the majority of participants from the sponsoring institution.

Future studies of CR leadership development should continue to focus on the concrete, intermediate to long-term impact of course participation. Additional evaluations of the impact of such leadership programs (eg, by addressing questions like: Were participating CRs' programs improved? Was learning by trainees in these programs demonstrably enhanced? Did quality and patient safety improve in the CRs' programs?) will enhance current understanding of the value of such activities and provide opportunities for future research.

Conclusions

This leadership development course for CRs was associated with self-reported enhancement of leadership competencies, both in immediate and intermediate time frames. It was acceptable to the participants and the PDs who recommended them.