Integrating Post-Operative Feedback into Workflow: Perceived Practices and Barriers

Abstract

Objective

Previous studies have found that both resident and staff surgeons highly value post-operative feedback; and that such feedback has high educational value. However, little is known about how to consistently deliver this feedback. Our aim was to understand how often surgical residents should receive feedback and what barriers are preventing this from occurring.

Design

Surveys were distributed to residents and attending surgeons. Questions focused on the current frequency of post-operative feedback, desired frequency and methods of feedback, and perceived barriers. Quantitative data were analyzed with descriptive statistics, and text responses were examined using coding.

Setting

University-based general surgery department at a Midwestern institution

Participants

General surgery residents (n=23) and attending surgeons (n=22) participated in this study.

Results

Residents reported receiving and staff reported giving feedback for procedure specific performance after 25% versus 34% of cases, general technical feedback after 36% versus 32%, and non-technical performance after 17% versus 18%. Both perceived procedure-specific and general technical feedback should be given over 80% of the time, and non-technical feedback should happen for nearly 60% of cases. Verbal feedback immediately after the operation was rated as best practice. Both parties identified time, conflicting responsibilities, lack of privacy, and discomfort with giving and receiving meaningful feedback as barriers.

Conclusions

Both resident and staff surgeons agree that post-operative feedback is given far less often than it should. Future work should study intraoperative and post-operative feedback to validate resident and attending perceptions such that interventions to improve and facilitate this process can be developed.

Introduction

Common to many skill-based professions, surgical training follows an apprentice-based model, where resident surgeon competency is generally measured by case volume and subjective evaluation from surgical staff (1, 2, 3, 4, 5). With changes to resident duty hours, there is concern surrounding resident competency at the end of clinical training (6). Work-hour limitations require residents to learn a large breadth of knowledge in a limited time, requiring efficient teaching. Common methods of surgical teaching include feedback, textbooks, simulation, and classes aimed to developing specific skills (7, 8, 9, 10, 11, 12).

Many of these education methods are utilized in surgical training. Recently, they were integrated into training curriculums to address the formalized six core competencies; patient care, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice (6). These competencies continue to evolve as newer forms of assessments develop that provide resident surgeons with objective evaluation. While these educational methods can be employed at various points surrounding a surgical case; it appears that feedback, a component of practice-based learning and improvement, may remain one of the few methods available to provide prompt and immediate education in the operative environment (10).

In the medical field, feedback provides trainees with a means of assessment to guide and improve future performance (14). As such, feedback should be provided quickly and efficiently, with specific points to correct improper decision-making and clinical skills (15). In the university setting, residents receive operative feedback to sharpen technical and decision-making skills. Residents and staff see the benefits to feedback; however, incongruity exists between the amount and the quality of feedback provided (16, 17). While studies have identified the importance of feedback, little work has explored how often residents and staff believe feedback should be given and what barriers they perceive to providing and receiving feedback exist in the current operative environment. The purpose of this study is two-fold. We aim to understand how often resident and staff surgeons perceive post-operative feedback should occur, and what perceived barriers are preventing feedback from occurring.

Materials and Methods

Setting and Participants

This study was performed within a general surgery department at a 566-bed academic medical center located in the Midwest. General Surgery residents rotate through a tertiary care center, a children’s hospital, and veteran’s hospital with full-time general surgery staff members supervising in all locations. Surveys were collected during a one month period, with residents and attendings given the opportunity to opt out of the study. The University of Wisconsin Hospital and Clinics (UWHC) Institutional Review Board granted this study an education exemption and approved this study.

Survey Development

A survey for two groups—one for residents and one for staff—was created and circulated amongst the research team consisting of two staff surgeons, one resident surgeon, and a medical education expert. The team, who had prior experience in creating and distributing surveys, used consensus agreement to develop a final version of the survey (18, 19) (Appendix 1 and 2). After development, the survey was piloted on ten laboratory residents and two staff members to test ease of completion and ensure used terms (i.e. operation-specific feedback, general surgery skills, and non-technical performance) were easily interpretable. Due to continued inquiry during the pilot, non-technical skills was further clarified by providing examples of communication, teamwork, and leadership. Resident surgeon survey questions addressed the current perceived occurrence of feedback post-operatively, the desired occurrence and methods of feedback, perceived barriers to feedback improvement, the desired setting to provide feedback, and demographics including post-graduate year (PGY). Staff surgeon survey questions covered reciprocal questions of the same content, except for demographics questions which covered surgical specialty and years in practice.

Research protocol

The surveys were distributed in October of the academic year to all clinical general surgery residents and staff. Data collection occurred for a total of one month. Research general surgery residents were excluded as they have minimal clinical experience during their research years and they were used during the pilot process. Surveys were distributed to the entire faculty who work with general surgery residents on a regular basis at the university hospital, children’s hospital, and Veterans Hospital. Attending surgeons who practice primarily at the community hospital location were excluded as they regularly work with off-service residents. Residents and staff were informed that their participation was completely voluntary and anonymous. Both groups were encouraged to fill the surveys out with no particular consideration of the rotation they were on. Rather, residents and staff were asked to consider what the most ideal situation would be across all operative experiences. If any questions arose, participants were encouraged to email or call study investigators for further clarification.

The survey was distributed to residents using an internet based platform (Qualtrics, Provo, Utah), while a paper version was distributed to the attendings and then later coded into Qualtrics. This choice was made to distribute internet surveys to residents and paper surveys to attendings given historic response rates to surveys at the institution. Careful attentions was paid to keep the format the same in both paper and internet versions. Accreditation Council of Graduate Medical Education (ACGME) operative logs were retrieved for each PGY.

Data Analysis

Responses were analyzed using Qualtrics and SPSS (Chicago, IL) where descriptive statistics, t-tests, and ANOVA analyses were performed. Text responses from open-ended questions were analyzed using open coding (19). We utilized descriptive coding in order to identify recurring themes within the resident and staff responses. T-tests were performed to determine any differences between experience-level (resident versus attending) on feedback type. A three-way ANOVA was performed to examine the effect of experience-level (resident versus attending), phase (current versus desired), and feedback type on feedback frequency. An ANOVA was also run to examine for differences in response by PGY.

Results

A total of 51 surveys were sent out to both resident and staff surgeons. Forty-five surveys were completed in total (response rate = 88%). The survey was sent to 27 residents, with 23 residents completing the survey (response rate = 85%). Resident experience ranged from PGY 1 to PGY 5 year (M= 2.8 years, SD= 1.3 years, 61% female). The number of operative cases varied by year: PGY 1 residents performed an average of 144 cases, PGY 2 performed an average of 256 cases, PGY 3 performed an average 293 cases, PGY 4 performed an average of 277 cases, and PGY 5 residents performed an average of 315 cases. A total of 24 staff were sent the survey, with 22 of 24 completing the survey (response rate = 92%). Surgeons experience ranged from 1 year to 30 years (M =9.3 years, SD = 7.8 years, 29% female). All staff surgeons were in an academic practice of various surgical sub-specialties, including: Endocrine, Colorectal, Trauma and Acute Care, Oncology, Hepatobiliary, Minimally Invasive, and Pediatric.

Assessing perceived current feedback

Resident surgeons reported receiving and staff surgeons reported providing feedback similarly overall (Figure 1). Residents and staff similarly identified general technical feedback occurring after an average of 34% of cases (SDres= 36%, SDsurg= 32%). Feedback on non-technical skills (eg. communication, teamwork, leadership) was also similarly reported; both groups reported receiving or providing feedback on non-technical skills after 17% (SDres=18%) and 15% (SDsurg=12%) of cases, accordingly.

Figure 1.

Resident and staff surgeon responses to current and ideal feedback practices

The greatest disagreement between resident and staff surgeons occurred with procedure-specific feedback. Resident surgeons reported the occurrence of procedure-specific feedback after an average of 25% (SD =15%), while staff surgeons reported giving this type of feedback more often after an average of 34% (SD = 26%) of cases. However, there was no statistically significant difference between the responses (p>.05). Additionally, ANOVA did not demonstrate any differences in resident responses based on PGY (p > 0.05)

Areas for improvement

Residents and staff both recognized the importance of operative feedback, and that it should occur more often than it is currently occurring. Residents reported they should receive general technical feedback at an average of 82% of cases (SD=22%) with staff similarly reporting an average of 87% of cases (SD=25%). Procedure-specific feedback was also identified as feedback that should be provided more often. Resident surgeons felt they should receive procedure-specific feedback after an average of 84% of cases (SD=19%). Staff surgeons reported analogous feelings, feeling procedure-specific feedback should be provided at an average of 81% of cases (SD=27%).

Both groups agreed that feedback should be given less often on non-technical skills than feedback related to general technical and procedure-specific skills. Residents reported they should receive non-technical skill feedback in an average of 57% of cases (SD=29%). Staff responses were similar, reporting they should provide non-technical skill feedback at an average of 49% of cases (SD=28%). Three-way ANOVA analysis revealed no significant interaction between experience, current versus desired frequency of feedback, and feedback type (p > 0.05). Additionally, ANOVA did not demonstrate any differences in responses based on PGY (p > 0.05)

Overwhelmingly, both resident and staff surgeons agreed that providing feedback verbally at the end of the operation in the OR was the most ideal method and setting (Figure 2). Residents ranked receiving written feedback soon after the operation, or included with end of rotation feedback as equal second-best options. Staff, however, preferred end of rotation feedback slightly more as a secondary method, with written feedback soon after the operation as less preferable. Other methods written in by residents or staff included during the case, or after through dictation or video review.

Figure 2.

Resident and staff surgeon responses to ideal feedback methods.

Barriers to feedback

Barriers to feedback reported by residents and staff included time, competing responsibilities at the end of the operation, inappropriate environment, and lack of interest. Time was reported most by residents (71%) and staff (88%) as the predominant barrier to giving and receiving feedback (Table 1). Competing end of case responsibilities was also frequently identified, with 38% of residents and 73% of staff discussing this. The operating room being an inappropriate environment was reported as an additional barrier to feedback by residents (29%) and staff (53%).

Table 1.

Most common barriers to feedback mentioned by resident and staff surgeons

Code	Examples mentioned by residents	Examples mentioned by surgeons
Time	“Time, need to move forward with case quickly and efficiently to keep the OR running smoothly.” “Don’t see them until next case [or] later in the day”	“Too little time in the day overall to meet after each case to review.” “Time constraints”
Competing end of case responsibilities	“Attending leaves OR to talk to family while we are closing…” “I feel too many times the attending is not present until just when the case is about to start and then they leave when we close…so you have lost any opportunity to discuss things”	“I let the resident close with the med student and then go talk to the family.” “Case flow makes it difficult…to provide appropriate feedback in a timely fashion” “Need to go talk to family every case and [it] doesn’t seem right to make them wait just to give feedback”
Inappropriate environment	“It is difficult to give and receive feedback with an audience.” “Environment (number and type of individuals in the operating room)”	“Too difficult for most residents…in OR, particularly in front of RNs, anesthesia.” “Other people are present…[or there is] no private place to do so”
Lack of interest	“Interest on the part of the attending.” “Desire to get other work done [or] move on to the next patient” “Unwillingness to spend time on feedback”	“In the middle of the night…both parties are not in the mindset to give/receive feedback” “Resident’s minds may not be receptive to feedback”

Discussion and Conclusions

This study surveyed resident and staff surgeons’ opinions of post-operative feedback and its perceived barriers. When corresponding resident and staff questions were analyzed, statistical analyses revealed no significant difference in opinion, suggesting that both groups hold similar opinions on the importance of post-operative feedback.

Feedback and Non-technical Skills

The general trend of these results indicate that the current practices of providing feedback are significantly different than what residents and staff perceive as being ideal practice. Both groups agree that more instances of post-operative feedback should be provided that assess procedure specific, general technical, and non-technical skills. While both groups desire procedural and general technical feedback greater than 80% of cases, non-technical feedback appears to be less valued in the post-operative setting at approximately 50%. Based on resident and staff responses, the general perception is that feedback ought to focus primarily on how to improve technically in surgical performance. This fails to recognize, however, the importance of non-technical skills in surgical practice, often referred to as the hidden curriculum.

Recent surgical literature on operative performance emphasizes the importance of teaching resident surgeons both non-technical and technical skills (20). Importantly, a failure of non-technical skills in the operating has been shown to lead to adverse patient outcomes (20). Scoring systems have been developed in response to this concern, measuring communication, leadership, and teamwork skills (21, 22, 23, 24). The goal of developing these scoring systems was to objectively measure non-technical skills, identify interventions to improve them, and ultimately, improve patient outcomes. Given the importance that non-technical skills holds on patient outcomes, we were surprised that less importance was placed on this when requesting feedback. Participating residents and staff may have placed lesser emphasis on non-technical skills for multiple reasons: (1) non-technical skills were not heavily emphasized while staff were receiving feedback as residents, (2) surgical literature has only recently noted the importance of non-technical skills, and (3) providing feedback for non-technical surgical skills may be more difficult than technical surgical skills. It also may be due to the lack of understanding on the value of non-technical feedback. In order to better emphasize the importance of non-technical skills, residents and staff alike could be further educated on the importance of assessing non-technical skills and its association with patient outcomes. Furthermore, staff could be trained on how to assess and provide feedback for non-technical skills.

Feedback and Clinical Workflow

When questioned about the best practices of feedback, residents and staff feel feedback is best provided verbally, immediately after the operation. However, when both groups listed perceived barriers, they felt that the additional personnel (or team members) in the operating room made it an inappropriate environment to provide meaningful, and often critical, feedback.

Other perceived barriers identified by both groups included time and differing post-operative obligations. As a case transitions to close, resident and staff responsibilities are time sensitive and differ considerably; both groups reported that residents frequently close the patient, put in post-operative orders, and escort the patient to the post anesthesia care unit. Concurrently, staff break scrub to debrief with the patient’s family, meet the patient of the next case, or complete other administrative work. These differing obligations restrict the opportunity for feedback in the post-operative setting.

Given the strong agreement between staff and residents that post-operative feedback is an important practice, however, it follows that post-operative workflow needs to change to accommodate this crucial teaching moment. Survey responses demonstrate that in the post-operative setting, work is being performed with the end goal of providing patient care. Perhaps one of the greatest reasons that post-operative feedback is still neglected is that it is not readily regarded as an aspect of patient care. This notion in surgical culture must change in order for staff and residents to integrate post-operative feedback into daily clinical practice. Specifically, staff and residents need to appreciate that post-operative feedback could provide residents with critical teaching points that lead to the overall improved patient outcomes. Given that both parties believe that providing said feedback verbally is considered adequate, it appears conceptually that such a process would take a minimal amount of time while maximizing patient benefit. With this crucial first step of accepting post-operative feedback as a part of the post-operative workflow, other barriers that are less difficult to solve, such as the lack of private environment to provide feedback, could then addressed in greater detail.

Feedback and Surgical Education

Recent changes to the American Board of Surgery requiring general surgical residents to undergo multiple operative assessments during their training has increased the use and availability of tools available for helping to assess residents’ performance and guide feedback. These include the Operative Performance Rating System (OPRS) from Southern Illinois University, which has been validated as a more comprehensive and shorter, single-item assessment, as well as a comprehensive assessment tool from the University of Wisconsin that evaluates procedure specific, general technical, and non-technical operative skills (25, 26, 27). Other studies have attempted investigating different methods of debriefs, such as the Procedure Feedback Form that incorporates self-assessment by the resident and evaluation by staff and a study from the Mayo Clinic integrating both post-operative and team feedback prior to closing the patient (28, 29). However, no work has been done to gauge the success of such interventions in comparison to formal post-operative feedback, nor do these interventions address the desire to have feedback in a private setting.

Interest in understanding intraoperative education coincided with the increased regulation of resident duty hours. Studies have analyzed both resident and staff surgeon interactions (30) and content of their communications (31), the role of “war stories” in intraoperative education (32), and how resident education may be influenced by the operative environment (33). Improving the educational methods will not only make surgical education more efficient, but also has the potential to increase residents’ surgical abilities and competencies (34). Providing surgical residents with knowledge and experience that address the core competencies will require improving education in all aspects of resident education (35, 36) including, but not limited to, feedback. Through understanding the current state of various education methods, we can refine and enrich intra- and post-operative education in the future.

Feedback in a Complex Work Environment

Since the introduction of “To Err is Human” (37), research in healthcare quality has become a mainstream focus. Many times, healthcare issues that are well known are poorly solved due to the complexity of the healthcare work place (38). For example, even though many patient safety issues have been identified, little progress had been made to actually solve them (39, 40, 41). A strongly held belief is that the complexity of the health care work place does not lend itself to simply a qualitative or quantitative research approach; rather it is best served by a mixed methods approach which boasts the advantage of analyzing multi-faceted work related issues in its native setting (42, 43). Human factors and ergonomics (HFE) research has grown impressively in the healthcare field, realizing the importance of employing a mixed methodology and successfully analyzing complex healthcare issues (44, 45, 46, 47). Given that our particular study aimed to understand and improve upon the intricacies of a complex work environment, we elected to utilize mixed methodology research, where analysis took place in both a quantitative and qualitative fashion (48). Furthermore, a mixed methods approach would best equip us to analyze the wide variety of data captured given the complexity of the operating room. Our ultimate goal was to maximize the breadth and depth of this complex patient safety and surgical education topic.

While this study addressed the aims stated, there were a number of limitations. The feelings expressed by both residents and staff are reflective of only one department in one academic institution. Future work should evaluate varying departments across multiple teaching institutions, including academic and community teaching programs. Interviewing a larger number of programs of varying background would not only improve the generalizability of the findings in this research, but also increase the understanding of potential differences in barriers based on type of program. Additionally, this study did not address specific methods for delivering post-operative feedback or specific content that should be covered. Further work including interviews or focus groups may provide additional insight into understanding the methods of ideal feedback. Finally, while this study tries to capture the current and ideal practices of feedback in surgery, it fails to assess what the perceived quality of current feedback is versus the ideal quality of feedback.

This study successfully demonstrated that resident and staff surgeons share similar opinions on receiving and providing feedback as it presently exists. Both groups believe feedback should occur more frequently. Changing the understanding of the value of post-operative feedback could optimize resident education, better prepare residents for independent surgical performance, and optimize patient outcomes.

Appendix 1. Post-operative survey administered to general surgery residents

Attached is a short questionnaire about post-operative feedback, which is part of our study on curriculum interventions to improve surgical education. The purpose of the research project is to improve the educational efficiency and effectiveness of residents’ clinical training. We estimate the questionnaire will take 1–3 minutes to complete.

Your participation in this research study is voluntary. If you do choose to participate, please return your completed form to your administrative assistant, who will then return them to us to help maintain your anonymity.

If you have any questions or concerns, please feel free to contact us at any time.

• 1What PG year are you? ___ 1 ___ 2 ___ 3___ 4___ 5
• 2What is your gender? ___ Male ___ Female
• 3How often HAVE YOU RECEIVED operation-specific feedback from the attending after completing a case together?

  Please give your estimated percentage: ______

• 4How often do you THINK YOU SHOULD receive operation-specific feedback from the attending after completing a case together?

  Please give a percentage: ______

• 5How often HAVE YOU RECEIVED feedback on general surgical skills from the attending after completing a case together?

  Please give your estimated percentage: ______

• 6How often do you THINK YOU SHOULD receive feedback on general surgical skills from the attending after completing a case together?

  Please give a percentage: ______

• 7How often HAVE YOU RECEIVED feedback on non-technical performance (e.g. communication, teamwork, leadership) from the attending after completing a case together?

  Please give your estimated percentage: ______

• 8How often do you THINK YOU SHOULD receive feedback on non-technical performance (e.g. communication, teamwork, leadership) from the attending after completing a case together?

  Please give a percentage: ______

• 9What do you see as barriers to giving feedback in the operating room?
• 10What do you think would be the best way to receive feedback on performance in the operating room? Please rank from best (1) to worst (4).

  ______ Verbally, at the end of an operation, in the OR

  ______ Written, soon after the operation

  ______ With the end of rotation feedback

  ______ Other (please explain): _____________________________________________

• 11Please let us know if you have any other comments or thoughts about operative feedback below:

Appendix 2. Post-operative survey administered to general surgery attendings

Thank you for agreeing to participate in this survey regarding the educational efficiency of the clinical teaching environment. We are working on a study to identify areas of resident clinical training that can be made more efficient to improve the overall educational value of residents’ clinical training. To help identify areas for improvement we are asking attending surgeons to respond to the following questions about post-operative feedback. All responses are entirely confidential and anonymous. Please answer all questions based on your experience operating with residents during your time as a faculty member here at UW.

• 12What is your specialty? ______________________________________________________
• 13How many years have you been in practice? ____________
• 14What is your gender? ___ Male ___ Female
• 15How often DO YOU GIVE operation-specific feedback to the resident after completing a case together?

  Please give your estimated percentage: ______

• 16How often do you THINK YOU SHOULD provide operation-specific feedback to the resident after completing a case together?

  Please give a percentage: ______

• 17How often DO YOU GIVE feedback on general surgical skills to the resident after completing a case together?

  Please give your estimated percentage: ______

• 18How often do you THINK YOU SHOULD provide feedback on general surgical skills to the resident after completing a case together?

  Please give a percentage: ______

• 19How often DO YOU GIVE feedback on non-technical performance (e.g. communication, teamwork, leadership) to the resident after completing a case together?

  Please give your estimated percentage: ______

• 20How often do you THINK YOU SHOULD provide feedback on non-technical performance (e.g. communication, teamwork, leadership) to the resident after completing a case together?

  Please give a percentage: ______

• 21What do you see as barriers to giving feedback in the operating room?
• 22What do you think would be the best way to give feedback on performance in the operating room? Please rank from best (1) to worst (4).

  ______ Verbally, at the end of an operation, in the OR

  ______ Written, soon after the operation

  ______ With the end of rotation feedback

  ______ Other (please explain): _____________________________________________

• 23Please let us know if you have any other comments or thoughts about operative feedback below:

  Thank you for participating in our survey!