SURGERY CLERKSHIP EVALUATIONS DRIVE IMPROVED PROFESSIONALISM

Abstract

PURPOSE

To determine whether a brief student survey can differentiate amongst third-year clerkship student’s professionalism experiences and whether sharing specific feedback with surgery faculty and residents can lead to improvements.

METHODS

Medical students completed a survey on professionalism at the conclusion of each third-year clerkship specialty rotation during academic years 2007-2010.

RESULTS

2007-2008 comparisons of survey items revealed significantly lower ratings for the surgery clerkship on both Excellence (F = 10.75, p < 0.001) and Altruism/Respect (F = 15.59, p < 0.001) subscales. This data was shared with clerkship directors, prompting the surgery department to discuss student perceptions of professionalism with faculty and residents. Post-meeting ratings of surgery professionalism significantly improved on both Excellence and Altruism/Respect dimensions (p < 0.005 for each).

CONCLUSIONS

A brief survey can be used to measure student perceptions of professionalism and an intervention as simple as a surgery department openly sharing results and communicating expectations appears to drive positive change in student experiences.

COMPETENCIES

Professionalism, Interpersonal and Communication Skills

INTRODUCTION

Professionalism is a concept frequently discussed in medical education,¹ including its importance in surgical training.^2-4 In general, professional attitudes and behaviors in medicine are described according to three domains: 1) provider characteristics (e.g., respect, integrity, and accountability), 2) ethical integrity and 3) sensitivity to the unique issues of each patient.⁵ The Liaison Committee on Medical Education (LCME) requires that student learning experiences be evaluated to maintain appropriate standards of professionalism.⁶ Two challenges face those tasked with ensuring that medical student experiences include training in professionalism. First, a brief, easily administered instrument is needed to assess specific aspects of professionalism in student learning environments. Second, effective methods are needed to improve professionalism among clinical faculty and residents contributing to medical student education.

A variety of instruments have been used to index components of professionalism,^7-13 however, most of these have been applied at either the institutional or training program level and have not been used to assess discrete medical student education experiences, i.e., clerkships within an institution. The Professionalism Survey developed by Arnold, Blank, Race, and Cipparrone is a brief evaluation tool validated for postgraduate training which has reliable dimensions corresponding to the main attributes of professionalism.¹⁴ We adapted this survey to measure the climate of professionalism in third-year medical student rotations.

Studies indicate medical professionals learn and come to internalize professionalism primarily through role modeling¹⁵ so we used a validated instrument to measure whether attendings and residents across required clerkships consistently modeled professional behaviors and attitudes. Since this instrument allows identification of specific behaviors associated with professionalism, we planned to also use the results to inform faculty development.

METHODS

Survey Instrument

The Clerkship Professionalism Survey (Table 1) lists 12 statements, each with a six-point Likert agreement response scale (strongly disagree [1], moderately disagree [2], somewhat disagree [3], somewhat agree [4], moderately agree [5], strongly agree [6]). This survey is identical to the original Professionalism Survey¹⁴ except that the phrases “residency training” and “residency colleagues” were deleted and the attribution of “residents” was changed to read “residents or attending physicians.” Survey items are grouped in three subscales: Excellence, Honor/Integrity, and Altruism/Respect. The initial validated Professionalism Survey¹⁴ phrases some questions in a manner such that a score of 1 reflects a more professional behavior than a 6. In order to alter the validated tool as little as possible, the Clerkship Professionalism Survey keeps the original wording and reverse scoring is used to report results for those items. The reverse scoring translates the results so that higher scores consistently reflect more professional behaviors.

Table 1.

Clerkship Professionalism Survey^*: Survey Items and Subscale Reliability

Student Rating Scale: strongly disagree [1], moderately disagree [2], somewhat disagree [3], somewhat agree [4], moderately agree [5], strongly agree [6]	Original* alpha	OHSU** alpha
Excellence	0.72	0.63
During this clinical rotation, I have met individuals whom I consider role models. My colleagues who are residents or attending physicians have assisted me in attaining educational materials pertaining to patients I am seeing. I observed that the residents or attending physicians I have worked with educate their patients about their illness. During my most recent clinical rotation, I have encountered individuals who display and promote professional behavior. I have observed resident or attending physician colleagues place the needs of their patients ahead of their own self-interest.
Honor/Integrity ***	0.60	0.57
I have been urged by my resident colleagues or attending physicians to copy their history and physical exam rather than gather my own information from the patient. I have been instructed to withhold data from a patient’s chart without being given an explanation from a senior resident or attending physician. I have observed a resident colleague or attending physician lie to a patient. The residents or attending physicians I have worked with have asked me to write orders or fill out forms and sign their names to them.
Altruism/Respect ***	0.59	0.71
I have observed residents or attending physicians making derogatory comments about other medical/surgical specialty groups or other health care workers. I have observed residents or attending physicians referring to patients as “hits, gomers, real citizens, walkie-talkies, players, frequent flyers” or other terms. I have observed residents or attending physicians scheduling tests or performing procedures at times that are more convenient for themselves than for the patient.

^*Adapted from Arnold EL, Blank LL, Race KEH, Cipparrone N. Can professionalism be measured? The development of a scale for use in the medical environment. Acad Med. 1998;73(10):1119-1121.

^**Oregon Health & Science University

^***For reporting, these survey items are Reversed Scored so that a higher score is desirable and reflects more professional behavior.

Survey Administration

Third-year students at Oregon Health & Science University (OHSU) rotate through seven required clinical rotations: child health, family medicine, internal medicine, obstetrics/gynecology, psychiatry, rural primary care, and surgery. For over a decade, OHSU has required students to complete a standardized evaluation of their learning experiences across clerkships prior to receiving their grades. During the 2007-2008 academic year, OHSU augmented this online evaluation with the Clerkship Professionalism Survey questions at the conclusion of every required clerkship; students were thus obliged to also complete the questions focused on professionalism to receive their grades. This resulted in a nearly 100% return rate. Data was collected for the 2007-2008, 2008-2009, and 2009-2010 academic years. Responses were anonymous, and only survey completion was tracked at an individual level. The OHSU Institutional Review Board approved this study under the exemption for research involving normal educational practices conducted in established or commonly accepted educational settings. Clerkship Professionalism Survey questions to be added to student evaluations of clerkship learning experiences were shared with and approved by clerkship directors prior to being instituted. There was no general announcement of the addition of these questions to any clerkship faculty, thus faculty were not aware they were being rated in these areas until after 2007-2008 survey results were shared.

Survey Analyses

Statistical analyses were carried out using SPSS, version 19. To facilitate interpretation, certain items were reverse-scored so that higher scores consistently indicated positive professional behaviors. Comparisons of scores from the first and second half of the 2007-2008 academic year revealed no seasonal differences or maturation effects across the year. Thus, all surveys for each academic year were combined and used for the analyses. The subscale scores of Excellence, Honor/Integrity, and Respect/Altruism were calculated by summing items and dividing by the number of items included in each subscale. Reliability was determined using Cronbach’s alpha scores. Subscale reliability measures of the survey used with third-year medical students were comparable to those of the original survey of postgraduate trainees. We compared mean ratings across clerkships for the subscales and individual items with ANOVA, using post-hoc Scheffé tests. A similar analysis was used to assess subscale scores across years within clerkships. Because of the multiple comparisons assessed, the Bonferroni correction adjusted the alpha levels to the p < 0.01 level.

Changes Driven by Survey Findings

Findings from the 2007-2008 academic-year surveys were presented to all clerkship directors during one of their monthly meetings held in late 2008. The survey findings indicated that although overall mean subscale scores (range 4.2 – 5.9) indicated that students had observed professionalism to some degree on all clinical rotations, the surgery clerkship received lower ratings than any other clerkship on all subscales. In addition, significant differences were found between surgery and other clerkships on two subscales and several individual items.

In response to these survey findings, the Department of Surgery undertook specific activities. The surgery clerkship director discussed the survey results with the surgery department chairman and education committee, which prompted faculty discussions and raised departmental awareness of student perceptions of surgery faculty and professionalism. Subsequently, the surgery education committee shared the survey results and a written summary of faculty discussions about its findings with all surgery faculty and residents. This written communication acknowledged the positive attitudes of the surgery faculty and the department’s reputation for its friendly atmosphere and noted some possible explanations for the survey’s lower ratings. However, the communication also stressed that, whatever the reason for the lower ratings, changes were necessary to improve clerkship students’ perceptions of surgery’s professionalism.

Following this, a portion of a department-wide meeting agenda with faculty and residents was dedicated to a discussion of the professionalism survey results. The group addressed specific survey items, subscale scores, survey validity, potential explanations for the scores and possible student biases regarding the surgery clerkship due to specialty stereotypes. Issues discussed included the hierarchical structure of surgical training, how comments may be interpreted negatively from less experienced viewpoints, maintaining professional behavior in stressful situations, and exhibiting consistent respect when speaking of other physicians and specialties. Department leaders explicitly stated that they expected to see improvements in student perceptions of professional behavior during the coming academic year.

This first professionalism meeting was held during the latter part of the 2008-2009 academic year with surgery faculty and nearly 90 residents during their weekly protected education time. The discussion is now repeated annually during the surgical resident orientation, with additional time devoted to how residents might best model professional behavior and interactions while teaching, evaluating, and working with medical students. Each year the 2007-2008 study results are used to emphasize the importance of student perceptions. In addition, professionalism issues are now explicitly addressed by the surgery clerkship director with third-year students at the beginning of every surgery clerkship orientation.

RESULTS

A total of 734 surveys were assessed for the 2007-2008 academic year and were compared with results from 868 surveys from 2008-2009 and 432 surveys during the six months of data collection during 2009-2010. All students completed surveys at the conclusion of each clerkship since survey completion was required to obtain a grade.

The 2007-2008 mean survey scores for the subscales of Excellence, Honor/Integrity, and Altruism/Respect for each clerkship are presented in Figure 1. Although subscale comparisons revealed no significant differences among the seven clerkships on the Honor/Integrity subscale, significant differences were found between surgery and the other clerkships on the Excellence (F = 10.75, p < 0.001) and Altruism/Respect (F = 15.59, p < 0.001) subscales. On the Excellence subscale, surgery was rated significantly lower than the family medicine, internal medicine, and rural primary-care clerkships (p < 0.001 for each). On the Altruism/Respect subscale, at baseline the variance was even more striking with surgery rated significantly lower than all six other clerkship experiences (p < 0.001 for each).

Figure 1.

2007-2008 Third Year Clerkship Professionalism Survey Subscale Ratings

Surveys continued to be collected in 2008-2009, in which the surgery clerkship continued to be an outlier in terms of student perceptions of professionalism. However, during the third academic year of the Clerkship Professionalism Survey, 2009-2010, (see Figure 2) the mean scores of the surgery clerkship significantly improved while there were no significant changes in mean scores for any of the other clinical rotations. On both the Excellence and Altruism/Respect subscales, differences observed between surgery and other clerkships in 2007-2008 were no longer present. In 2009-2010 the mean scores on both the Excellence and Altruism/Respect subscales for the surgery clerkship were significantly higher (p < 0.005 for each, See Figure 3); this change occurred after the after the surgery department meeting on professionalism.

Figure 2.

2009-2010 Third Year Clerkship Professionalism Survey Subscale Ratings

Figure 3.

Comparison of the Surgical Clerkship Excellence and Altruism/Respect Subscale Scores Across Three Academic Years

DISCUSSION

This study has informed our ability to improve medical student education in professionalism in several ways. A clerkship professionalism survey can be used to identify particular aspects of professionalism and to discriminate among students’ clinical experiences on different clerkship rotations. Sharing results of such a survey can identify specific areas for faculty and resident growth in professionalism. Finally, increasing awareness of student perceptions of faculty and resident professionalism, in the context of supportive departmental leadership, can be an effective way of promoting faculty self-reflection of their personal behaviors, how they are perceived by others, as well as how they can improve their role modeling of professionalism during students’ clinical experiences.

The results of this study also confirm the psychometric properties of the Professionalism Survey and demonstrate its ability to be adapted to assess the professional climate of clerkship as well as residency experiences. Our findings also provide potential calibration for expected scores within required third-year clinical rotations. Evaluations of professionalism are typically conducted at the institutional level or within single training programs.^7-13 We found that this instrument can be used within a medical school to discriminate different clerkship student experiences of professionalism but also can drive departmental improvements in faculty and resident professionalism. In addition, we found the instrument to be useful in documenting professionalism both within and across clerkships over time. Because student professionalism issues need to be addressed and carefully documented by faculty, this anonymous instrument would not be useful for use in the individual evaluation of student professionalism. However, the Clerkship Professionalism Survey could be adapted to a variety of clinical and educational settings where the intent is to measure the overall climate of professionalism experienced by learners.

While the survey itself was valuable, we believe the manner in which the survey data was presented to clerkship directors was key to the improvements that followed. In addition to identifying differences among experiences, the individual survey items provided specific feedback regarding professional behaviors observed by students during their clerkship experiences. We found that openly presenting clerkship directors with data on their own and other clerkships’ ratings on specific behaviors provided ample impetus for positive change. Due to the pride teaching physicians commonly take in their own specialties, no department wishes to be perceived as “less professional” than other specialties. Although actual differences (though statistically significant) between ratings were small, the open sharing of data prompted, in this case, faculty and resident awareness of student perceptions as well as behavioral changes which significantly improved subsequent evaluations of the professional climate of the surgery clerkship. Whether these behavioral changes were due more to physician self-reflection and a sincere desire to practice more professionally, by departmental pride and directives, or simply by the knowledge that students were evaluating them on professionalism is not known—likely with most faculty and resident behavior changes were due to a combination of all of the above. Based on significantly higher surgery scores on the 2009-2010 subscales, the awareness generated by sharing survey data from students with surgical faculty and residents appeared to influence student perceptions and experiences of surgery clerkship professionalism. While formal faculty development workshops are often utilized to address professionalism,^16,17 we found that a single meeting, simple in scope and brief in duration, proved to be an effective intervention in this case. This meeting allowed surgery faculty and residents to dialog about possible methods to dispel surgical stereotypes, address how workload stresses can affect professional attitudes and behaviors, and review how relatively inexperienced learners may view comments and behaviors differently than colleagues.

The original development of the survey on professionalism was prompted partially by findings that learners frequently heard derogatory comments regarding patients, colleagues and other health professionals.¹⁸ In our study, the Altruism/Respect subscale of the survey indexed those behaviors, and it had the greatest initial difference across clerkships. Its items included behaviors that have been termed “badmouthing” or “bashing.”^19,20 While perhaps used as a means to vent frustrations and cope with stress, such remarks or “gallows” humor may have unintended adverse effects on trainees and others in the environment.^{21,22, 23}

Improving student experience of and, thus, training in professionalism may be accomplished most effectively through department-wide awareness and priorities. This is the dimension most associated with the “hidden curriculum” in which enacted social norms can supersede written objectives.²⁴ The surgery department’s combined department-wide discussion, to acknowledge challenges and establish shared responsibility for upholding new behavioral expectations, was an important component of improving student experiences in the area of professionalism. Clearly, a leadership that regularly emphasizes and demonstrates the importance of professionalism is key to successful, sustained improvement. Students are generally more influenced by what they observe than by stated professional standards,¹⁸ and, thus, altering the day-to-day culture of the surgical clerkship was implicit to the improvement in student ratings.

Limitations

Although the survey items describing specific behaviors of interest²⁶ have subscale reliability²⁵ and face validity, survey responses are based upon student recollections and estimations rather than a direct audit of experiences. In addition, students may begin the third year having little direct experience with different specialties and may be influenced by cultural stereotypes.²⁷ As a result, findings may have been affected by confirmation bias due to student expectations of different experiences.²⁸ Directly addressing and putting professional issues in context during the surgery clerkship student orientation may also have positively impacted surgery professionalism ratings. However, improvement in scores reflecting observations of specific behaviors by faculty and residents suggests that the professional environment of the surgery clerkship was indeed favorably altered.

CONCLUSIONS

The interest in physician professionalism is growing because of its relationship to quality of care and health outcomes.²⁹ Recently the challenges of providing training in professionalism and the need for new methods of training have been highlighted.³⁰ Other findings suggest learners can provide formative information that can guide targeted faculty development in professionalism.³¹ We found that anonymous assessment of student perceptions can not only identify specific professional behaviors observed in different learning contexts, but that sharing survey results can also drive improvements in medical education and practice. Effecting change may not require much more than motivating faculty and residents through brief discussions and clear messaging from leadership. The findings of this study are useful for those responsible for the professional education of medical students, residents, and faculty.