Evaluation of inter-professional communication and leadership skills among graduating Canadian urology residents

Abstract

Introduction

The importance of developing inter-professional communication and leadership skills among residents is well-recognized; however, formal tools to assess these skills are lacking. The goal of our study was to assess the leader and communicator roles in graduating urology residents using a validated self-assessment form developed for business students that focuses on inter-professional skills.

Methods

Chief residents (n=36) were evaluated with surveys of communication and leadership skills. The same surveys were administered through email to the residents’ program directors (PDs). Resident self-assessment and PD assessment were compared using paired and non-paired t-tests.

Results

Graduating urology residents’ self-assessment of their communication and leadership skills did not differ from assessments made by their PDs (77.6 vs. 74.4%; p=0.19); however, there were outlier candidates in whom PD assessment differed substantially from self-assessment on both surveys. Graduating urology residents scored themselves higher on self-awareness (82.6 vs. 77%; p=0.05) and lower on stress management (67.7 vs. 77%; p=0.01) compared to their PDs. Resident self-assessment scores were similar to business students on both communication and leadership surveys. Limitations were the small sample size and lack of survey evaluation by those surveyed.

Conclusions

Graduating urology residents’ self-assessment of their own communication and leadership skills did not differ greatly from assessment by their PDs or a sample of business students. Comparison of self-assessment evaluations and evaluations by PDs allowed us to identify outliers in whom self-assessment and PD assessment markedly differed, which may allow for more focused and meaningful feedback.

Introduction

The importance of communication and leadership skills, in addition to other non-clinical skills, is well-recognized. The shift towards a global competency as opposed to merely mastering medical knowledge and skills prompted the Royal College of Physicians and Surgeons of Canada’s (RCPSC) Health and Public Policy Committee to create the seven CanMEDS domains: Medical Expert, Professional, Health Advocate, Scholar, Manager, Collaborator, and Communicator in 1996. More recently, the Manager domain has been replaced by the Leader domain.¹ These roles and competencies have been adopted by major accreditation bodies worldwide. In the U.S., the Accreditation Council for Graduate Medical Education (ACGME) has also recognized the importance of non-clinical skills and includes communication within the core competencies that all residents must achieve.²

Residents are expected to communicate effectively with a wide range of medical professionals and to lead/manage teams of medical students, more junior residents, and allied health professionals on a daily basis. They are also required to communicate in a timely and effective way with attending staff about patient care. While some tools have been developed to evaluate the communication and leadership skills of medical professionals with patients, there is a lack of evaluation tools available for inter-professional communication and leadership skills.³ A survey study of graduating urology residents in Canada found that only 10–30% had been exposed to formalized communication assessment tools throughout their residency.⁴ A survey of program directors across Canada showed satisfaction with their evaluation of the Medical Expert role, but poor satisfaction with assessment of the other CanMEDS competencies, including Communication and Leader roles.⁵ Therefore, there remains a need for formal tools to assess inter-professional communication and leadership skills among residents.

Within the business world, evaluation tools aimed at inter-professional communication and leadership skills have been developed. The Personal Assessment of Management Skills (PAMS) and Supportive Communication surveys are readily available surveys that have been previously validated on 5000 business students.⁶ These surveys consist of 84 (PAMS) and 20 (Supportive Communication survey) statements for which candidates are asked to rate how strongly they agree with each statement according to a six-point Likert scale. These surveys take approximately 10–15 minutes each to complete, making them concise enough to be practical for use by busy clinicians. The goal of our study was to evaluate inter-professional leadership and communication skills among graduating urology residents using these validated self-assessment forms and to compare the residents’ self-assessment to assessment by their program director (PD). By comparing self-assessment to PD evaluation, we aimed to identify residents who may benefit the most from targeted feedback, a process used in multisource feedback called gap analysis.⁷

Methods

Subjects

We surveyed graduating urology residents at the time of the annual Queen’s Urology Examination Skills Training Program (QUEST) in Kingston, Ontario. We chose these participants as a convenience sample. Approval was obtained from the Queen’s University institutional review board with assurance of confidentiality provided to all participants. The same surveys were administered through email to the residents’ PDs.

Surveys

The Supportive Communication survey consisted of 20 statements grouped into the following themes: coaching/counselling, providing negative feedback, and supportive communication. Candidates were asked to rate how strongly they agreed with each statement according to a six-point Likert scale. ⁶ The numbers (1–6) corresponded to “strongly disagree,” “disagree,” “slightly disagree,” “slightly agree,” “agree,” and “strongly agree,” respectively.

The PAMS survey consisted of 84 statements grouped into the following themes: self-awareness, managing stress, creative problem-solving, supportive communication, gaining power/influence, motivating others, managing conflict, empowering/delegating, teamwork, and leading positive change. Candidates were asked to rate how strongly they agreed with each statement according to the same Likert scale described above.

Gap analysis

As no standard method for performing gap analysis within multisource feedback has been established, we used reference quartiles from the previously published data of 5000 business students.⁶ Residents were deemed outliers if self-assessment differed from PD assessment by two or more quartiles from the reference business student group.

Statistics

Mean ± standard deviation (SD) resident self-evaluation and PD evaluation score were reported as a percentage. The residents’ self-response score and PDs’ evaluation score were compared using both paired and unpaired student’s t-test. A two-sided p value of <0.05 was the threshold for statistical significance in all comparisons. Analyses were performed using GraphPad Prism 7.

Results

Chief residents (n=36) were evaluated with two surveys at the time of a weekend course. The response rate was 100%. The response rate from PDs was 54%, accounting for 33% of the residents evaluated (n=13).

Communication skills

On the communication survey, the mean percentage for resident self-assessment was 77.6% (SD 6.2), which was not different than the score from their corresponding PD (mean 74.4±9.9; p=0.19) (Fig. 1). Evaluations by both residents and PDs were similar to self-evaluations reported by a sample of business students (mean 75.5%). Fig. 2 shows a paired comparison of resident self-evaluation and PD evaluations with business student quartiles marked by dotted lines for reference. Although mean resident and PD scores were not significantly different (p=0.57), there were two candidates in whom self-assessment differed from PD assessment by two or more quartiles from the reference business student group.

Fig. 1.

Mean percentage scored on communication (A) and leadership (B) skills survey of residents’ own self-assessment and assessment by their program directors (PDs). Dotted lines indicate quartile scores of 5000 business students.

Fig. 2.

Paired scores of resident’s self-evaluation and assessment by their program directors (PDs) for communication (A) and leadership (B) skills. Dotted lines indicate quartile scores of 5000 business students.

On sub-analysis of communication skills, there were no differences in evaluation of coaching/counselling, providing negative feedback, or supportive communication skills among residents and PDs (Table 1).

Table 1.

Residents’ self-assessment of their own communication skills compared to program directors’ assessments

	Residents n=36	PDs n=12	p
Coaching/counseling	77.6 (8.6)	75.5 (11)	0.48
Giving negative feedback	76.3 (9)	72.9 (9.3)	0.27
Supportive communication	78.7 (7.5)	77.1 (10.1)	0.58

PD: program director.

Leadership skills

On the leadership (PAMS) survey, the mean percentage for resident self-assessment was 76.5% (SD 6.48), which was not different than the score from their corresponding PD (mean 76.2±10.3; p=0.9) (Fig. 1). Paired comparison of evaluation scores among residents and PDs was also not different (p=0.94) (Fig. 2). Although mean resident and PD scores were not significantly different, there were four candidates in whom self-assessment differed from PD assessment by two or more quartiles from the reference business student group. Evaluations by both residents and PDs were similar to self-evaluations reported by a sample of business students (78.3%).

On sub-analysis of leadership skills, residents rated their self-awareness skills higher (82.6 vs. 77; p=0.05) and stress management skills lower than their PDs (67.7 vs. 77; p=0.01). There were no differences in evaluation of problem-solving, supportive communication, gaining power, motivating others, managing conflicts, delegating, teamwork, and leadership skills among residents and PDs (Table 2).

Table 2.

Residents’ self-assessment of their own leadership skills compared to program director’s assessments

	Residents n=36	PDs n=10	p
Self-awareness	82.6 (6.6)	77 (11.4)	0.05
Stress management	67.7 (9.6)	77 (8.7)	0.01
Problem solving	74.3 (8.4)	72 (11.6)	0.55
Supportive communication	78.2 (7.9)	76 (13.5)	0.47
Gaining power	75.4 (11)	79 (7.1)	0.4
Motivating others	74.9 (9.1)	74 (11.3)	0.73
Managing conflicts	83.1 (7.8)	78 (11.6)	0.13
Delegating	79.2 (8.6)	81 (9.3)	0.58
Teamwork	78.5 (9.2)	75 (11.7)	0.3
Leadership	79.3 (8.1)	77 (9.4)	0.46

PD: program director.

Discussion

The importance of developing inter-professional communication and leadership skills among graduating residents is well-recognized; however, formal tools to assess these skills are lacking. In this study, we used inter-professional communication and leadership self-assessment forms previously validated among business students and administered these to residents and their PDs to obtain a multisource appraisal of inter-professional communication and leadership skills among graduating urology residents. We found that graduating urology residents’ self-assessment of their own communication and leadership skills did not differ from assessment by their PDs overall, however, there were some outlier residents in whom self-assessment scores differed markedly from PD evaluation. Resident self-assessment scores were similar to business students on both communication and leadership surveys.

We obtained both resident self-evaluations and evaluation by PDs to obtain a multisource assessment of resident communication and leadership skills. Multisource feedback consists of evaluation by oneself and others, including supervisor(s), subordinates, and colleagues.^8–11 Multisource feedback has been shown to have high reliability, validity, and feasibility in the business world and more recently has been used in medical education.^9–11 One of the biggest advantages of multisource feedback is the possibility for gap analysis. Gap analysis consists of a systematic comparison between the scores obtained from a colleague/mentor and the self-evaluation of the participant being assessed.⁷ Although paired comparison of resident and PD evaluations were not statistically different, there were two residents on the communication survey and four residents on the leadership surveys in whom the self-evaluation differed from the PD evaluation by two or more quartiles. By detecting areas in which participants over- appraise or under-appraise their abilities, gap analysis enables mentors to provide focused feedback in needed areas, which can better allow for meaningful change.

Other techniques to evaluate non-clinical skills among residents have been used with variable success. In orthopedic surgery and neonatology, efforts have been made to develop objective structured clinical examinations (OSCEs) to assess the Communication, Professional, and Collaborator roles.^12,13 However, interstation reliability was low in these studies for the Manager/Leader and Communicator roles.¹³ The cost of using simulated patients for OSCEs can also be prohibitive.¹² Literature has shown that the ideal evaluation/survey length is between 10 and 20 minutes.^14,15 Candidates were able to complete each survey in approximately 10–15 minutes. The ease of administration, focus on inter-professional skills, and use of multisource feedback is a large advantage of the assessment technique used in the current study over techniques such as OSCEs and in-training evaluation reports (ITERs).

In contrast to the Medical Expert role, Communicator and Leader CanMEDs roles represent more implicit characteristics that are by nature more difficult to teach. A previous survey of graduating urology residents showed that only 36% had formal communication training available to them from their urology program, core curriculum, or postgraduate department.⁴ However, there is evidence that communication and leaderships skills can be learned. Use of standardized checklists and providing communication training during surgical handover were shown to lead to improved communication and decreased adverse events.¹⁶ There is also evidence that leadership development programs among physicians can lead to improved self-assessed knowledge and even some improvements in quality indicators for disease management.¹⁷ To build leadership skills among urologists, a recent publication by Beiko et al discussed current leadership techniques and training sources, including the Canadian Medical Association leadership course, the American Urological Association leadership program, and various books.¹⁸ As Vince Lombardi said, “Leaders aren’t born, they are made. And they are made just like anything else, through hard work.”

Looking at characteristics assessed within the leadership survey, graduating urology residents scored themselves higher on self-awareness (82.6 vs. 77%; p=0.05). Interestingly, the statements regarding self-awareness, “I seek information about my strengths and weaknesses from others as a basis for self-improvement” and “In order to improve, I am willing to be self-disclosing to others (that is, to share my beliefs and feelings),” had the highest discrepancy among residents and PDs, with residents reporting higher agreement and PDs reporting lower agreement. Inherent in the role of PD is the duty to provide regular feedback to residents throughout residency. As adult learners, the resident is tasked with seeking feedback on strengths and weaknesses throughout residency. The discrepancy of the statements described above may indicate that residents feel they seek evaluation from PDs more often they actually do.

Graduating urology residents scored themselves lower on stress management (67.7 vs. 77%; p= 0.01) compared to their PDs. Statements regarding stress management in the survey refer specifically to time management skills, maintaining a calm demeanor at work, and maintaining interests outside of work. One interpretation is that residents are able to project a demeanor in the work place that is calmer than they actually feel.

Strengths of this study are that we used previously validated, widely used, and widely available communication and leadership surveys that focus on inter-professional skills. Another strength is the high response rate among graduating residents. Aweakness of this study is that we did not ask graduating urology residents or PDs to evaluate the surveys as a feedback tool; however, evaluation of the survey tool was not our main objective, as this was a descriptive study. The relatively low number of participants is also a weakness of our study. Unfortunately, in a small specialty like urology, the sample size for survey studies is going to be relatively small. However, we did have a response rate of 100% of the residents and 54% from PDs. We feel the response rate from PDs is adequate, given the demanding schedules of most PDs and the fact that most PDs had multiple chief residents to evaluate, making the survey more burdensome to respond to compared to the residents. Additionally, our multisource feedback was limited to self-evaluation and PD evaluation only, whereas optimal multisource feedback includes feedback from underlings and peers.

Conclusions

The importance of developing inter-professional communication and leadership skills among graduating residents is well-recognized; however, formal tools to assess these skills are lacking. We used communication and leadership self-assessment surveys previously validated among business students and showed that graduating urology residents’ self-assessment of their own communication and management skills did not differ greatly from assessment by their PDs. Comparison of self-assessment evaluations and evaluations by PDs allowed us to identify outliers in whom self-assessment and PD-assessment markedly differed. By detecting these areas in which participants over- appraise or under-appraise their abilities, faculty can provide focused feedback, which may allow for more meaningful change in communication and leadership skills among residents.