Annual Performance Reviews Of, For and By Faculty: A Qualitative Analysis of One Department’s Experiences

Abstract

Purpose:

Although annual performance reviews and feedback are recommended for faculty development, best practices and faculty perceptions have not been documented. The authors sought to evaluate the process in one medical school department that established and has sustained an innovative review tradition for 25 years.

Method:

Content analysis of faculty reports and immersion/crystallization to analyze interviews.

Results:

Faculty reports described satisfaction and dissatisfaction; facilitators and barriers to goals; and requests for feedback, with community, collaboration and mentorship integral to all three. Interviewees emphasized practical challenges, the role of the mentor and the power of the review to establish community norms.

Conclusion:

Respondents generally found reviews constructive and supportive. The process informs departmental expectations and culture.

Introduction

Annual performance reviews have been a mainstay of human resources practices (Buckingham & Goodall, 2015; Greenfield, 2015; Heskett, 2006). The opportunity to evaluate and enhance performance has outweighed some limitations of reviews, such as the time-intensive process and the risk of bias (Schoenberger, 2015). Reviews provide motivation, contribute to job satisfaction, establish group norms, and enhance effective leadership (Quast, 2013). The impact of annual performance reviews for academic medical faculty has not been assessed to date.

At its inception, the founding chair (TSI) of the Harvard Medical School Department of Population Medicine established an annual ‘review of progress’ with several goals that included broadening everyone’s knowledge of other’s work and communally discussing faculty career progress to inform mentors’ feedback to mentees. This annual review has endured and the medical school has identified it as a model for other departments. Our objective was to describe this annual review and to determine whether the intentions of the department’s leadership and faculty were realized. We conducted a qualitative study in which we analyzed two types of data: extant faculty members’ self-reports provided for the review and semi-structured interviews of a subset of faculty at each rank.

Method

Population and Setting

Founded in 1992, the Department of Population Medicine (originally known as the Department of Ambulatory Care and Prevention) at the time of this study (academic years 2008–09 to 2011–12) included faculty with MDs, research doctorates, or both. During the study period, forty-five faculty participated in the review process: 25 women and 20 men; 34 white, 10 Asian-American and 1 African-American. Table 1 describes the distribution by rank each year. Of the 38 faculty reviewed who were not professors, 24% were promoted once during the 4-year study interval.

Table 1.

Faculty distribution by rank, academic years 2008-09 through 2011-12

	2008-09	2009-10	2010-11	2011-12
Instructor/Lecturer	12	14	11	12
Assistant Professor	9	11	11	8
Associate Professor	9	8	6	9
Professor	7	7	8	8
Total	37	40	36	37

The department is not governed by a tenure clock or ‘up or out’ system. Rather, faculty are reviewed annually to ascertain their eligibility for promotion; there is no set time at rank. The department chair proposes all promotions and reappointments to the medical school based on the results of the annual performance review meeting.

The Annual Review

The annual faculty review meeting is the culmination of several faculty development activities that include faculty meetings by rank, ‘strategy seminars’ to address key topics of relevance to promotion and academic success, and meetings for women faculty. In early spring, faculty members draft annual progress reports (Table 2a), discuss them with their mentors, and pose questions to be discussed at the review meeting. Progress reports, typically 2–3 pages in length, and CVs are shared with the entire faculty prior to the day-long meeting. The agenda is ordered by academic rank, beginning with fellows. With no fellows present, each fellow’s mentor, in turn, gives a brief presentation of the progress and issues discussed in the 1:1 meeting. Others who work with the mentee also provide input, followed by general discussion of the individual. After all fellows have been discussed, the instructors and lecturers leave and each is discussed in turn, followed by assistant and then associate professors. Those present for each discussion include all faculty members at ranks above the group being discussed. Full professors are present for the entire day but are not reviewed. Topics discussed vary by individual and can include readiness for promotion, funding challenges, skills development (e.g. presentation or writing skills, oversight of mentees), suggestions for collaborators, career path, personal concerns, and recognition of successes. Following the annual review meeting, the mentor-mentee pairs meet again, and mentors provide a summary of the group discussion to their mentees.

Table 2:

a. Self-report form questions b. Interview questions

a. Self-report questions:

1. Highlight your major professional achievements during the past academic year.

2. Did you achieve your goals this past year? Comment on significant factors influencing your performance. What were the catalysts in achieving your goals? What were the barriers? What can the department do to help you to achieve your goals in the upcoming year(s)?

3. Outline your professional goals for the upcoming year. Be specific and include where appropriate research (grant submissions, publications, professional presentations), teaching (where and for whom, supervision pre/postdocs), administrative, leadership, service and clinical work (where, how much).

4. What questions or issues are most important to you for feedback from the departmental annual review?

b. Interview questions:

1. Could you describe briefly your experiences with the annual review process since joining the department as a faculty member? What have been the good or most useful aspects? What have been the disappointing or negative aspects? Are there topics you do not feel comfortable raising for discussion at annual review?

2. I’d like to learn a little bit about your relationship with your departmental mentor(s).

3. How have you and your mentor discussed the annual review, before and/or after the meeting?

4. Could you describe your perceptions of the annual review process from the point of view not only of someone who has been evaluated but as a member of the community evaluating others? For example, we would be interested in how you perceive the discussions about those junior to you.

5. Are there any other aspects of the departmental annual review process that you feel are worth commenting on?

Evaluation of the annual review process

We chose to conduct an open-ended qualitative study to delve into how the faculty used and experienced the annual review rather than to impose a priori assumptions or hypotheses. Thus, we performed conventional content analyses of the faculty’s written annual self-reports to uncover dominant themes within the comments that faculty chose to share with the community. We undertook thematic analyses of interviews conducted by two of the investigators with a subset of faculty to understand their experience of the process. The Institutional Review Board of the Harvard Pilgrim Health Care Institute approved this study.

Procedure

In 2013, we emailed all current departmental faculty at the instructor, assistant, associate and professor ranks, informed them of the study aims, risks and benefits, and offered each the opportunity to opt out of this analysis; only one faculty member did so. We compiled all participating faculty members’ annual self-reports submitted 2009–2012, and identified 17 who 1) were not among the study investigators, and 2) had 4 consecutive years of data during this period. The study assistant assigned each faculty member a code and de-identified each report.

Conventional content analysis of self-reports

Conventional content analysis was used in three steps: 1) deconstruction (initial open coding), 2) interpretation (establishing meaningful categories), and 3) reconstruction (capturing the principal metaphors in the text as a global theme) (Hsieh & Shannon, 2005).

1. Deconstruction. To begin coding across the report queries (Table 2a), three investigators reviewed the self-reports of two faculty members, generated codes, discussed and reconciled differences until reaching common understanding. With this process, we identified 8 codes (numbers after each code represent the number of excerpts so coded): recognition of facilitators and barriers to meeting one’s goals (156); community and collaboration (128); academic products (119); leadership (82); feelings (58); mentorship (44); request for advice or feedback (43); and transitions and balance (35). We allowed double coding of excerpts: i.e., if a quote reasonably fit in two codes, we attributed it to both. Using these codes, the same three investigators coded the remaining 15 self-reports: each coded 6 reports, 4 alone and 2 duplicating another person’s coding (i.e., A+B, A+C, B+C), thereby continuing to check for agreement and to reconcile differences. We used Dedoose (http://www.dedoose.com/), a secure, web-based, qualitative research tool to code, quantify and sort text into appropriate categories and to review each other’s work.

2. Interpretation. Next, we grouped meaningfully related codes into four inductive categories: community, collaboration and mentorship; feelings of satisfaction and dissatisfaction; facilitators and barriers to success; and requests for feedback and advice. While the latter two categories align with two report queries (#3, #4, Table 2a), all codes and categories were found throughout the reports. Since lists of academic products contributed nothing to our understanding of how faculty used the reports beyond the obvious, we omitted that code.

3. Reconstruction. Through comparison of the categories, we determined that “community, collaboration and mentorship” was not distinct from, but integral to, the other three. Therefore, we identified one final overarching theme: community, collaboration and mentorship; and three sub-themes: feelings of satisfaction and dissatisfaction; facilitators and barriers to success; and requests for feedback and advice.

Structured interviews

In 2014–2015 we randomly selected 8 current faculty from the eligible 17, a male and female from each academic rank, to participate in structured interviews; all agreed to participate. The authors identified key questions of interest with the goal of understanding the perceived benefits and risks of the annual performance review experience, from the perspective both of being evaluated and of doing the evaluation (Table 2b). Two of the investigators conducted in-person 1:1 interviews, which were recorded and then transcribed by an independent, professional transcription service. Interviews lasted approximately 30 minutes. Using the immersion/crystallization technique two investigators independently conducted a thematic analysis of the transcripts by highlighting and annotating important sections for discussion, discussing these annotations, resolving differences of opinion where they surfaced and arriving at a consensus interpretation of important themes (Crabtree & Miller, 1999; Glaser & Strauss, 1999). Four major themes emerged: practical considerations, mentorship, focus of the review and community ethos (Table 5).

Table 5.

Faculty experiences of the annual review: semi-structured interviews.

Practical considerations	Mentorship	Focus of review	Community ethos
“… when I find my calendar more and more cluttered it's harder and harder to find time to do the pre-work, but once I get to the day then I'm really glad that I've done it.”	“People make comments or ask questions that are helpful for me to bring back to my mentee…”	“Whenever the person is smart enough to ask the right questions, they always get discussed very well.”	“I do come away every year thinking about how this is all about grants and publications. …it indicates what the heart of the DPM is.”
“It's tiring, but it's worth it.”	“It might be nice if it weren’t so mentor dependent… if there were other channels to get the feedback to the person.”	“The things that are emphasized are those things which are conducive to promotion.”	“It's a generous culture.”
“And to maintain the level of interest, and feel like you want to think about each individual, it just gets harder.”	(the review) “it adds a layer of objectivity, and adds some standardization across everybody”	“I thought that the goal of this, and sometimes it happens, is really to look at the person, and what’s right for that person.”	“Department is very proud of the process.”
			“I think it probably builds community.”
			“It’s nice to know about the breadth of - - and often depth of what people are doing.”

Results

Analysis of Self-Reports

Faculty Members’ Use of the Self-Report

Throughout the reports, faculty revealed their feelings, work habits, ambitions, and interests in the context of (a) their personal experience (Table 3) and (b) relationships with colleagues and departmental roles (Table 4).

Table 3.

Representative quotes from self-reports in key categories: Self perspective

Reasons for Feelings of Satisfaction & Dissatisfaction	Facilitators & Barriers to Success	Requests for Advice & Feedback
Research Productivity I feel pretty good about my accomplishments.	Time Management I discovered that by switching my schedule and coming in at 7am that I've been able to at least find a couple hours in the morning to do some substantive work…	Promotion What areas should I be working on for eventual promotion (aside from the obvious getting grants and publications) and suggestions for doing so, e.g. suggestions for raising my profile nationally, expanding my role as a mentor, advice on seeking committee and other leadership roles.
Funding I had been working on securing this funding for several years, so was gratified when it ultimately came through.	Balance I have been pretty selective about participating in these [visiting professorships] because they require travel and time away from both family and work.	Funding I’d appreciate advice on how to make my research …of greater interest to funders […]
Recognition for Work This was presented at a conference… and has received a fair amount of attention.	I planned to spend this year devoting more time to grant writing than manuscript writing…and time away from both family and work.	Efficiency Are there tasks I should be delegating or resources I should be taking advantage of to free up my time?
Meeting One’s Goals Not everything worked out as planned. My activities with … were disappointing […]	Between [a national] committee and work on [local] grants, I have let other goals, notably publication of first-authored work, as well as collaboration, editorial, and review responsibilities slip…	Balance Feedback regarding the balance between teaching and a research career emphasis would be helpful. How will I be able to determine the best balance between my leadership/administrative responsibilities for the Division…and my own research portfolio?
	Practical Assistance The major barrier for me this year was not having enough analyst time…

Table 4.

Representative quotes from self-reports in key categories: Community perspective

	Community, Collaboration & Mentorship
Perspective	Feelings of Satisfaction & Dissatisfaction	Facilitators & Barriers	Requests for Advice and Feedback
What community did, does or could do for the individual:	I feel extremely fortunate to be a member of our department. Thank you!	I largely achieved my goals, thanks to the support of colleagues.	I welcome creative ideas about how to align more effectively the goals of the [X] program with the goals of the department.
	As usual, guidance and help from my mentor and several other senior faculty members are critically important for me…	My project managers and grant managers have been amazing. Because they are so high functioning, I have been able to spend more time on research and less on administrative activities.
		In the upcoming years, I would like to be more involved in collaborative projects.
		As the department gets busier and busier, it would be nice to have another project manager or 2…
What individual did, does or could do for the community:	Continuing mentorship of junior faculty and fellows is a key component of my ongoing work.	The program’s connection to the department was strengthened through extensive faculty participation in resident teaching and in strengthening mentoring.	Guidance on how to keep my research in line with DACP’s missions and priorities while progressing within the department.
	Busy year with grant writing; worried about salary support for myself and research staff.	How can we make the teaching programs […] most helpful to the DPM faculty, consistent with our overall mission?	How to mentor fellows/students with such limited time.

A community ethos was especially evident through the tone of the reports, which was generally positive; positive expressions occurred three times more frequently than negative expressions. Faculty used words such as fun, happy, pleased, enjoy, appreciate, grateful, exciting and rewarding, but also worried, disappointed and sorry. Positive expressions focused on both the department (community, mentorship and collaboration – Table 4), “[Faculty members’ names] have been particularly generous in helping me find funding for programmers who have helped me become more efficient with my research” and the self (research productivity, receiving a grant, and recognition for one’s work – Table 3), “ I had been working on securing this funding for several years, so was gratified when it ultimately came through” whereas negative expressions focused primarily on the self (e.g., frustration in not meeting goals or not getting funding – Table 3), such as “My activities with the [project name] were disappointing.”

What the community did for the individual

The reports naturally reflected the fact that all faculty members had mentors and almost all worked on collaborative teams; but they also described a more extensive community of support, including administrative assistants, data analysts, and the department as a whole (Table 4). “I really appreciate [names] and others making continuous efforts to include me in …collaborative projects…I feel I am better connected both within and outside (department).” In asking for help, faculty members principally addressed two types of issues: professional outcomes (e.g., funding and academic promotion), “Is there still a concern that I have not achieved career independence as it relates to promotion? If so, how should I address this concern?” and professional development (working effectively and balancing competing demands), “Are there tasks I should be delegating…to free up my time and make my projects move along more efficiently?”

Faculty members’ concerns with balance were mostly professional, e.g., how to balance different components of an academic position. “Feedback regarding the balance between teaching and a research career emphasis would be helpful; specifically ways in which I can foster each without diluting both.” It was rare for faculty to request advice on work-life issues.

What the individual did for the community

Faculty expressed concern for the well-being of the department and their colleagues. They implicitly asked “What can I do for the department?” Faculty expressed a desire to mentor others and to align their projects/programs with the department’s mission. Some faculty wanted to fit into a team or initiate new department programs. In one instance, a faculty member changed the name of a program to associate it with, and to strengthen, the department’s mission. (Table 4), stating “How can we make the teaching programs…most helpful to the (department’s) faculty, consistent with our overall mission as a department?”

Analysis of Interviews

Practical Considerations

Some aspects of the review process proved challenging. The length of the review meeting increased with the growth in size of the faculty. The amount of material prepared (reports and CVs for every faculty member) was generally more than some could digest. Mechanisms for reviewing professors did not fit the hierarchical structure for the review meeting process, since only faculty senior to those being reviewed remained present for the discussion.

Mentorship

Much of the benefit perceived was driven by a sense of the quality of mentorship. For those whose interactions with their mentor were typically focused on day-to-day work, the annual review created an opportunity to consider long-range career issues. The opportunity of mentors to observe how other mentors presented their mentees and their own advice for the mentees was described by some respondents as educational. Faculty with a perceived suboptimal mentorship match reported receiving less useful feedback and sometimes none. Other comments included the inability to incorporate the views of same-rank or outside mentors, and that mentors’ advice after the annual review meeting didn’t seem to have been influenced by the group discussion.

Focus of the Review

Some observed that the annual review often addresses readiness for promotion, sometimes at the expense of other considerations such as work-life balance or long-term fitness for an academic career. Some interviewees, particularly senior faculty, felt the focus and content of the review had changed as departmental leadership and priorities evolved over the 25 years.

Community ethos

Most interviewees acknowledged that the annual review was a supportive community activity that is a hallmark of this department. One respondent noted “It’s a generous culture.” Respondents appreciated that the tone was constructive, conveyed genuine concern, and was focused on creating connections. Some felt pride in the process, even stating that the review was one of the elements that attracted them to the department, while others thought that the department might be too self-congratulatory regarding the process. All interviewees agreed that the annual review helps establish norms for the departmental culture, with one stating “I think it probably builds culture” and another that “It’s nice to know about the breadth of – and often the depth of - what people are doing.”

Discussion

For 25 years, the Department of Population Medicine has conducted an annual faculty review involving all members of the faculty community. In our evaluation of faculty members’ written reports and interviews, we found that the review, while not without its challenges, continues to serve its original aims and remains a powerful anchor of departmental culture. The interdependency of the faculty in collaborative research groups, mentor-mentee dyads, methodology interest groups, and in other associations, is fully explored and exposed in the review process, as is the concern faculty have for each other’s success and career trajectory. In annual reports, one person stated “I largely achieved my goals, thanks to the support of colleagues, including project managers, programmers, epidemiologists and statisticians. There is great synergy between … [different teams’ projects in the department].” In an interview, a faculty member echoed this sense of interdependency, stating “…we have to take a collective responsibility for one another (which) I think is…important.”

While on balance faculty expressed satisfaction, some also highlighted areas in which the review fell short. Inconsistencies in the communication of review recommendations, the limited ability of the review to address the concerns of faculty who fall outside the dominant cultural norms, and practical challenges of the day-long process were among the less successful aspects of the review for some. One interviewee stated “I do come away every year thinking about how this is all about grants and publications. Period, period, end paragraph. And if you are doing something other than that, or not exclusively focused on that, then you feel a little irrelevant to the process. And also, it just -- I mean it indicates what the heart of the Department is.” When the relationship between the mentee and mentor was suboptimal, the entire approach was of limited value and, in one case, a respondent felt it was destructive.

We have introduced modifications to the process in response to concerns raised through this study. To increase consistency of information-sharing, the departmental administrator now takes detailed notes during the review meeting which she sends to the individual, the mentor, and the department chair. Because a single mentor may not be able to provide both informed and impartial guidance, we created a career advisor program to allow faculty to choose a secondary mentor, typically from outside one’s central area of interest, to allow for additional guidance. (Blood et al., 2012; DeCastro, Griffith, Ubel, Stewart & Jagsi, 2014; DeCastro, Sambuco, Ubel, Stewart & Jagsi, 2013; Straus, Johnson, Marquez, & Feldman, 2013; Tsen et al., 2012). While the system allows for feedback from a large faculty group, it is entirely top-down. Therefore, we created an online 360-degree survey, available to the entire department, to provide feedback on any faculty member or fellow. This process now allows for peers or for staff, junior collaborators, or mentees to provide input.

Limitations

This study is limited by the fact that the investigators have all been associated with the department, past or present. Whether our interview respondents were willing to be entirely candid with colleagues is not clear. The other source of data – reports written for purposes of the annual review – is not completely inclusive of faculty concerns and questions. For example, none of the written reports addressed concerns about the quality of the mentorship itself, though several individuals raised this issue in the interviews. Because the interviews were conducted with a limited, random sample and we did not attempt to reach saturation, it is possible other opinions are not represented.

Implications for Future Research

Whether this process can be replicated in other environments has not been demonstrated, although three other institutions have informed the authors that they are implementing this approach (personal communication). We anticipate that effectiveness in other settings would be adversely affected by the size of larger departments. It is not clear whether the review in isolation from the faculty development processes in which we also engage would be successful. Little evidence exists to support best practices in evaluating faculty. While some respondents highlighted shortcomings and challenges, on balance, our findings suggest that an intensive and comprehensive review of all faculty in a group setting can contribute to individual faculty development and can create a sense of community norms and values.

Biographies

Maureen T. Connelly is an internist and assistant professor in the Department of Population Medicine at Harvard Pilgrim Health Care Institute. In her role as Dean for Faculty Affairs at Harvard Medical School, she oversees faculty promotions and appointments, institutional policies related to faculty, and multiple faculty development programs including a junior faculty fellowship and leadership course.

Thomas S. Inui is a Professor Emeritus of Medicine at Indiana University School of Medicine and the Joe and Sarah Ellen Mamlin Professor Emeritus of Global Health Research. A primary care physician, educator, and researcher, he previously held positions as head of general internal medicine at the University of Washington School of Medicine and the Paul C. Cabot professor and founding chair of the Department of Population Medicine at Harvard Medical School. A health services researcher, Dr. Inui’s special emphases in teaching and research have included physician-patient communication, health promotion and disease prevention, chronic disease management, primary care effectiveness, the social context of medicine, and medical humanities.

Emily Oken is a physician with clinical training in internal medicine and pediatrics. Dr. Oken’s research focuses on the influence of nutrition and other modifiable factors during pregnancy and early childhood on long-term maternal and child health, especially cardiometabolic health and cognitive development. Dr. Oken’s commitment to research mentorship has been recognized with a mid-career development grant from NICHD and a Young Mentor Award at Harvard Medical School. Within the Department of Population Medicine, she is Vice-Chair, Director of Faculty Development, and site director for the Harvard General Medicine fellowship. At Harvard Medical School she is Associate Director and Advisor in the Oliver Wendell Holmes Society.

Antoinette Peters is a corresponding member of the faculty of the Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute. Prior to retirement, she was associate professor of the Department of Population Medicine and associate director of the Center for Teaching and Learning, the Academy at Harvard Medical School. Her research focuses on a variety of issues related to medical education and primary care.