Feedback on feedback: a two-way street between residents and preceptors

Abstract

Background

Workplace-based assessment (WBA), foundational to competency-based medical education, relies on preceptors providing feedback to residents. Preceptors however get little timely, formative, specific, actionable feedback on the effectiveness of that feedback. Our study aimed to identify useful qualities of feedback for family medicine residents and to inform improving feedback-giving skills for preceptors in PGME training program.

Methods

This study employed a two-phase exploratory design. Phase 1 collected qualitative data from preceptor feedback given to residents through Field Notes (FNs) and quantitative data from residents who provided feedback to preceptor about the quality of the feedback given. Phase 2 employed focus groups to explore ways in which residents are willing to provide preceptors with constructive feedback about the quality of the feedback they receive. Descriptive statistics and a thematic approach were used for data analysis.

Findings

We collected 22 FNs identified by residents as being impactful to their learning; analysis of these FNs resulted in five themes. Functionality was then added to the electronic FNs allowing residents to indicate impactful feedback with a “Thumbs Up” icon. Over one year, 895 out of 8,496 FNs (11%) had a “Thumbs up” added, divided into reasons of: confirmation of learning (28.6%), practice improvement (21.2%), new learning (18.8%), motivation (17.7%), and evoking reflection (13.7%). Two focus groups (12 residents, convenience sampling) explored residents’ perception of constructive feedback and willingness to also provide constructive feedback to preceptors.

Conclusion

Adding constructive feedback to existing positive feedback choices will provide preceptors with holistic information about the impact of their feedback on learners, which, in turn, should allow them to provide more effective feedback to learners. However, power differential, relationship impact, and institutional support were concerns for residents that would need to be addressed for this to be optimally operationalized.

Introduction

There is a rapidly expanding international uptake of competency-based medical education (CBME).^1-5 One key principle within CBME is the importance of providing residents with quality feedback on their clinical performance, with the underlying premise that this is critical for residents’ development.^6-15 Quality feedback has been defined as criterion-referenced, timely, behaviourally-specific, actionable, context-specific and tailored to individual needs.^16-20 It has been shown to improve performance.²¹ With the onus on preceptors to provide feedback that impacts learning, it is critical that preceptors continually hone this important skill. Preceptors stand to benefit from feedback on their feedback and residents are well placed to provide it.

There are many resources available for preceptors to learn how to provide effective feedback. Articles, learning modules, and faculty development workshops all provide helpful foundational information about feedback provision.^22,23 There are concerns, however, that such generic offerings are not effective in changing behaviour.^11-13 Providing personalized specific feedback about assessment performance does appear to improve assessment practices.¹³ A multipronged approach to faculty development aimed at providing generic knowledge about feedback-giving, layered with personalized feedback about this skill, is likely a reasonable strategy.

Such personalized feedback to preceptors can happen in a variety of ways. A common approach is to provide periodic anonymized resident-generated summative assessments of performance. For anonymity reasons these are often held until enough assessments can be collated to mask the assessors,²⁴ often delaying receipt of that feedback. Anonymity and delay can render feedback less effective. For anonymity to be maintained, comments necessarily lack specificity. Just as feedback to residents is best done in a timely and specific way to be optimally effective, feedback to preceptors should be the same. The concept of dialogue in feedback has been highlighted in the literature. Feedback is ideally not a unidirectional linear process but an interactive conversation promoting self-reflection and opportunities for clarification and exchange between participants.^25,26 Herein lies a dilemma: dialogue, either face-to-face or electronic, compromises anonymity and anonymity compromises meaningful dialogue.

Are residents ready to engage in providing this non-anonymous, upward feedback to their preceptors? Preceptors are responsible for the assessment of residents which will impact residents’ advancement and overall success in their training program. This inherent power differential can inhibit resident’s willingness to provide non-anonymous feedback.^27-29

Despite this power differential, fruitful efforts have been made to engage residents in non-anonymous feedback with their preceptors. Fluit and colleagues reported on a collated survey derived from multiple residents’ anonymous input that resident representatives then reviewed with the preceptor and moderator.³⁰ Dudek and colleagues reported on a carefully designed system in which residents provided both formative and summative feedback to their clinical supervisors.²⁷ This was carried out in a small program characterized by longer term relationships between faculty and residents. Feedback-giving skills were taught before residents engaged in providing feedback to their preceptors. This study reported residents’ willingness to engage in this non-anonymous upward feedback however it was unclear if the study was reporting on the formative or summative feedback. Prolonged contact between residents and faculty, may increase the possibility of building more trusting relationships, which may or may not increase the likelihood of residents feeling comfortable giving more difficult constructive feedback to their preceptors. Both these studies involved residents giving summative feedback to preceptors with whom they had prolonged contact.

This study aims to add to the existing literature looking at residents providing non-anonymous, timely, formative feedback to their preceptors, granting preceptors an immediate way to gauge their performance. There are gaps in the literature. We have not found any articles focussed on providing preceptors with behaviourally-specific formative feedback addressing feedback-giving skills. We are unaware of studies where residents in large programs, with less preceptor continuity, provide non-anonymous feedback to their preceptors. Research is required in large programs to determine how residents can be supported in giving formative feedback to preceptors. Our study looks at providing feedback to preceptors in three new ways to address these gaps; 1) residents giving preceptors timely, behaviourally specific formative feedback on feedback-giving skills, 2) within a large program, and 3) with less resident-preceptor continuity. In addition, we also describe an electronic means for providing such timely feedback, critical in a large program for feasibility and practicality reasons. Constructive feedback is used throughout this paper to replace the term negative feedback as feedback focusing on changing behaviour (formerly called negative), if given in an effective way, is often very constructive.

The purpose of this study was to determine, from the resident’s perspective, (a) the most useful qualities of preceptor’s formative narrative feedback to promote resident’s learning, and (b) to determine willingness and ways to facilitate residents giving both positive and constructive formative behaviourally specific feedback to their preceptors about the feedback they receive. This is done in the context of a large postgraduate training program using an electronic assessment system.

Methods

Our research is an exploratory study which informs improving feedback-giving skills for preceptors in any PGME training program. It delves into ways to facilitate residents providing both positive and constructive formative feedback to their preceptors about their preceptor’s feedback-giving skills using an electronic assessment system, in a large family medicine program (140 residents), with hundreds of preceptors, and often limited contact time between residents and preceptors.

We adopted a two-phase exploratory design for this study. In Phase 1, we conducted a mixed methods exploratory sequential design where we began with collecting qualitative data from preceptor feedback given to residents through narrative comments on Field Notes (FNs; see context section for a description of FNs). We then collected quantitative data that identified the number of FNs where residents provided feedback to their preceptor about the quality of the feedback given. Informed by Phase 1, Phase 2 employed a qualitative exploratory approach through focus groups to further identify ways in which residents are willing to provide preceptors with constructive feedback about the quality of the formative assessments they receive.³¹ Ethics was obtained from our institution’s Health Sciences Research Ethics Board (File #: 6020068).

Settings and participants

This paper reports on research done between 2015 and 2017 within a large Canadian Department of Family Medicine postgraduate residency program in southern Ontario. The department includes approximately 140 postgraduate year (70 PGY-1 and 70 PGY-2) residents who work with many different preceptors. Residents have anywhere from 1 to 31 different preceptors writing FNs for them, the average being approximately 20 FN contributors/resident.

Context

This department embarked on a CBME approach in 2010. As part of facilitating the provision of more daily formative feedback to residents, the department developed electronic FNs. FNs are brief documents capturing the verbal feedback given by preceptors to residents.^32,33 In our context, FNs capture assessment of small aspects of performance, tied to the critical competencies needed for a developing family physician.³⁴ Within the FN, there is a qualitative narrative component wherein preceptors can reinforce what a resident has done well, and identify steps for further improvement. A dialogue between preceptor and resident about what is written in the FN is possible using the ‘comments box’ which is part of the FN interface. A complete description of our competency-based assessment practices is available.³⁵ Preceptors in the department have had access to numerous faculty development sessions and electronic modules focussed on feedback-giving. Academic Advisors in our system refers to one of the resident’s primary preceptors, who has the added role of meeting regularly with the resident throughout their training and is an integral part of making summative competency decisions.

Data collection and analysis

Phase 1 The first phase of this study identified helpful preceptor feedback-giving skills and generated an electronic means for residents to provide positive feedback to preceptors. In 2015, we held five FN competitions (with a $20 gift card as a prize), each involving 20 residents who were involved in their family medicine rotation. Over a two-week period, residents submitted any of their daily FNs that they found particularly helpful and described the reasons why. These friendly competitions were part of our change management strategy to enhance the amount and quality of feedback provision that our faculty were providing, a necessary part of our CBME implementation. Competition publicity served to highlight the value for residents of receiving quality feedback, and to reinforce qualities of good feedback, and so became informal opportunistic continuing medical education (CME)-type events. Through those competitions 22 FNs were submitted by residents as being most helpful for learning. Using an inductive, emergent design all submitted FNs underwent a thematic analysis.³⁶ We identified five themes of helpful feedback (see Table 1 and Appendix B). Subsequent FN competitions (held 2016-2019) and analysis of the submitted FNs confirmed these five themes; no additional themes emerged.

Table 1.

FN competition themes and descriptive results

TU FN Analysis 2016-2017 (n = 895)
Theme	Theme Selected (%)
Confirmed that I am doing the right thing	28.6%
My practice changed and improved	21.2%
Learned something new	18.8%
Motivated to learn more	17.7%
Promoted reflection	13.7%

In 2016, we added a ‘Thumbs-Up’ (TU) icon to the electronic assessment system. Residents, on reflecting about their electronically submitted FN, could opt to add a TU to their FN if they felt it was beneficial to their learning. Once they selected the TU icon, they could then choose from a drop-down menu one of the five behaviourally-specific themes to identify why this FN was particularly helpful. This was a first step in getting residents comfortable in providing feedback in a specific timely way to their preceptors. They could also add free-text narrative comments if they desired (see Figure 1). Figure 2 depicts feedback that a resident viewed as helpful for promoting reflection (Figure 2).

Figure 1.

An example of a FN that the resident is reviewing and is contemplating giving a TU (upper arrow). They have written a comment back to their preceptor about why they found this FN helpful (lower arrow).

Figure 2.

An example of a FN with a TU that indicates the theme ‘promoting reflection’

Preceptors receive information about their FNs in a timely way. We developed a reporting function within our electronic platform that allows each preceptor, the Program Director and Department Head access to a real-time collated report for that preceptor on the number of FNs they have done across competency domains, as well as the number of FNs given a TU with its associated theme. This positive feedback is done non-anonymously and so is available in real-time to preceptors.

From June 2016 to June 2017, a total of 8,496 FNs on our 140 residents were submitted. Descriptive statistical analysis was performed to identify the frequency with which residents gave their FNs a TU and which of the five themes they chose.

Phase 2: In the second phase of the study, we explored residents’ perceptions of providing constructive feedback for improving preceptors’ feedback-giving skills. Using convenience sampling, we recruited 12 end-of-year PGY-1 residents (4 male; 8 female) to participate in two semi-structured focus groups (n = 7, n = 5). Two researchers who are experts in qualitative research methods and not associated with the residents or their success in their training program conducted the focus groups. One researcher facilitated the discussion and the other wrote notes and summarized discussion points. Each focus group took place at a location convenient for participants, was one hour in length and audio-recorded.

First, we explored residents’ concerns about providing preceptors with constructive feedback. Second, by providing residents with sample constructive feedback statements, we explored their comfort with using these statements, revising those based on their input. Appendix A includes the questions asked in the focus group and the sample phrases of constructive feedback for preceptors. Finally, residents provided us with recommendations that would support residents providing preceptors with constructive feedback.

Both focus groups were completed using the same guide and the data from both were analyzed at the same time. No differences in the findings were identified between the two groups. The focus group interviews were transcribed verbatim. Data were analyzed through an inductive, thematic analysis approach. Two researchers independently open-coded both transcripts in NVivo 11,³⁷ and then met to compare codes to ensure intercoder reliability.³⁶ They discussed any discrepancies in coding and reached consensus to ensure shared meaning. Once a codebook was developed, the research team met multiple times to further discuss the codes and group them into similar categories and emerging themes. This process allowed our research team to mitigate personal biases and tensions, thereby addressing reflexivity and ensuring rigor in our process.^38-40 We verified all data to ensure consistency of our findings through member checking. We summarized the emergent themes at the end of the focus group for all participants to confirm.

Findings

We have organized the findings into two distinct parts. Part 1 describes the emergence of five themes from the analysis of the 22 FNs identified in the FN competitions. Part 2 identifies two overarching themes: (1) residents’ suggested wording and phrases for constructive feedback statements, and (2) residents’ concerns and recommendations for providing preceptors with constructive feedback.

Part 1. FN competitions: Themes of the Thumb-up FNs

Analysis of the 22 FNs submitted in the 2015 FN competitions identified five themes of preceptor feedback considered most useful to residents’ learning: (1) changed and improved practice, (2) imparted new knowledge, (3) provided motivation to learn, (4) confirmed skills, and (5) promoted reflection (see left column of Table 1 and Appendix A). By June 2017, 8,496 FNs were collected, of which 895 (11%) received a TU as feedback to preceptors. Distribution of these themes is illustrated in Table 1 (right column).

Part 2. Resident focus groups: Providing constructive feedback to preceptors

This section is organized into two distinct themes: (a) wording residents would be comfortable using in providing preceptors with constructive feedback, and (b) residents’ concerns about and recommendations for providing preceptors with constructive feedback

a) Wording residents would be comfortable using in providing preceptors with constructive feedback

At the focus groups, participants discussed four scenarios where preceptors’ feedback was not helpful: (1) no feedback at all, (2) generic feedback, (3) feedback that felt too negative, and (4) feedback that was too wordy. Participants scrutinized the sample phrases of constructive feedback statements. They identified and reconstructed the wording such that they would be comfortable giving this feedback to preceptors. These phrases could be added to the FNs for each of the scenarios mentioned below (Table 2). They explained that these statements should be constructed as being actionable by preceptors.

Table 2.

Constructive feedback statements for preceptors to improve feedback they provide residents to add to TU FN

If the feedback by preceptor...	Suggested phrases of constructive feedback to preceptors
Provides no feedback at all	-I am trying to work on improving my skills and some written coaching from you on this topic would be valuable to me. -I would really be interested to hear from you about what and how I can improve.
Generic feedback (No information on what or how to improve)	-I need to know how I can improve my skills. -I would love to hear if you have a better approach to this.
Feels too negative	-I would find it helpful if you could give me some coaching on how I can improve.
Is too wordy	-I sometimes get confused about what or how to improve. It might be best for me if your feedback to me is shorter.

I think some of them [preceptors] are very actionable, so if you tell them: ‘Ok, this particular feedback didn’t have enough information on how to improve my skills’. Then if there’s a way they can respond with like: ‘Here’s some additional points on how you can improve’ - it might allow for some response there. [FG1-R6]

Participants also preferred that the constructive feedback statement be phrased in a positive rather than negative manner, and be brief and straightforward.

I think #1 [sample statement] you’re saying “however” – it’s just the language. I think #2 and #3 are sort of more positive. Instead of saying that it was poor feedback, you’re saying that you’re just looking for a little bit more feedback as opposed to it being poor. So it’s the language in both of those are actually positive in saying that what you’ve given me so far was great and just need a little bit more. [FG1-R2]

b) Residents’ concerns about and recommendations for providing preceptors with constructive feedback

Focus group participants agreed that constructive feedback to preceptors would help preceptors make FNs more valuable for their learning. They, however, expressed concerns in providing this feedback. Those concerns were grouped into three themes: power differential between preceptor and learner, the impact on their relationship with their preceptor/Academic Advisor, and a lack of institutional expectations for preceptors to receive constructive feedback from residents. They also suggested recommendations to mitigate concerns (see Table 3).

Table 3.

Residents’ concerns about and recommendations of providing feedback to preceptors

Concerns	Recommendations
Preceptor-resident power differential	Keep feedback anonymous.
Preceptor/Academic Advisor-resident relationship	Collate and send all feedback to preceptor/Academic Advisor at the end of the year.
A lack of institutional expectation for preceptors receiving constructive feedback from residents	Provide preceptors with clear institutional expectations and support around feedback provision—both feedback to and from residents.

I. Preceptor-resident power differential: Participants acknowledged the power differential between residents and preceptors in residency training. In a learner position they felt uncomfortable telling their preceptors that their feedback was not very useful, concerned that constructive feedback to preceptors may impact their progression in the program adversely. Also, they acknowledged that having a longer-term relationship with the preceptor may lessen this concern:

…you feel there’s a huge power divide, in terms of your security net, if you say something that are you going to lose your residency spot or something like that. (FG2-R1)

I think having a more established relationship, feeling like it’s less of a power dynamic, feeling like you’re more established in your role as a resident, and maybe feeling like there’s going to be less repercussions. Because it shouldn’t be that big of a deal to say, ‘This feedback didn’t go over very well, or I can see where you are coming from but I think what you are saying didn’t have the intended effect’. But in my case things did go quite awry and I didn’t know how to address that at all. (FG2-R2)

Participants suggested that anonymity would be helpful to mitigate this concern and would encourage residents to provide constructive feedback to preceptors:

Alternatively, you could collect feedback from residents…into a large pool and distribute. Pick up themes from that…[and distribute] to all preceptors at the end or mid-year. So, that it is somewhat more anonymous. (FG1-R7)

II. Preceptor/Academic Advisor - resident relationship: Participants also expressed concerns that provision of constructive feedback may negatively affect their relationship with preceptors, particularly when the preceptor was also their Academic Advisor.

I don’t want it to impact the relationship. I guess part of our Academic Advisor meeting – one of the questions was, ‘Are you getting enough feedback?’ You can’t answer that honestly because your Academic Advisor is your preceptor…. Because you can’t say honestly without impacting your relationship. I find, at least that’s my fear. (FG1-R3)

To avoid negatively impacting preceptor/Academic Advisor-resident relationships, the participants suggested collating all feedback at the end of the block or year. In this way residents would keep a safe and comfortable relationship with preceptors and Academic Advisors during their training, and preceptors and Academic Advisors would learn from and reflect on constructive feedback:

All those responses could be collated and then given to the preceptor at the end of the year. That way your relationship with them is finished but they could learn every year. (FG1-R3)

III) Institutional expectation for preceptors receiving constructive feedback from residents: Participants felt that providing constructive feedback to preceptors and Academic Advisors was a new practice and it should be established in the residency training program. Once provision of constructive feedback by residents was felt to be an acceptable norm within the program, they would feel more confident doing this.

It makes a big difference if the preceptor or person giving the feedback opens them up or offers you that opportunity to provide feedback on their feedback. Because then they’re asking for it and sometimes they’re actually receptive to it as opposed to us trying to barge in and say, ‘This is feedback on your feedback’, and they are not receptive, they’re not asking for it. It makes a big difference. But, when the preceptors does that, it’s like, ‘Ok, this is the time and space and appropriate’. (FG2-R5)

If both residents and preceptors had clear expectations about the quality of feedback residents should receive, then residents would feel more comfortable asking for meaningful and useful feedback:

If there were more clear expectations for preceptors and like what they should put in a Field Note, then I would be more comfortable saying, ‘Oh, so we were supposed to have something like this every clinic, or, you were supposed to write a little bit more’. Instead of feeling like I’m subjectively demanding a little bit more from the preceptor. (FG1-R6)

Discussion

This study has shown that residents in a large program, with varying amounts of continuity with their supervisors, are willing to provide formative positive behaviourally-specific feedback to their preceptors about the quality of the narrative feedback their preceptor provided. This was done using a non-anonymous electronic system. Previous studies have reported on non-anonymous summative resident-derived feedback.^13,30 The study of timely non-anonymous formative feedback within a large program is a new addition to the literature. Given the evidence that providing personalized specific feedback does appear to improve assessment practices,¹³ this has potential to be a powerful form of faculty development. The timely formative nature of the TU feedback quickly identifies for preceptors the feedback residents find helpful and should reinforce these helpful feedback-giving behaviours.

Concerns have been raised that preceptors may not give constructive feedback for fear of a negative reaction or adverse preceptor assessment by the resident^{24, 41} and that residents may not attend to constructive feedback. Residents most often indicated liking feedback relating to performance assessed as being done well. Our study shows that residents were also appreciative (as witnessed by a TU designation) of constructive feedback that “teaches them something new” and “changes their performance”. This provides some reassurance that residents are not just choosing the TU as a reward for positive things said about their performance. It also reinforces that “negative” feedback is indeed constructive feedback and residents do appreciate it. The fact that only certain FNs are awarded a TU and that there is a good spread across the five themes shows that residents are discerning in the feedback they provide to preceptors about what they perceive to be helpful.

When asked to look at the flip side of feedback, that of providing constructive feedback to their preceptors aimed at improving their feedback-giving skills, residents endorsed, through their editing, wording that was behaviourally-specific and actionable and, importantly, that they would be willing to give to their preceptors. However, despite the professional nature of the wording agreed upon, residents still expressed concern about the mechanism for providing such constructive feedback. This discomfort primarily ties to their relationship with their preceptors and Academic Advisors (the power differential in the relationship, not wanting to impact their relationship with their Academic Advisor negatively). Residents felt that the effect of a power differential may be softened with a more established longer-term relationship between resident and preceptor. Practical suggestions were provided by residents to mitigate these concerns. Residents want a way to remain anonymous if providing constructive feedback. This option for anonymity reinforces what other programs^12,13 report, with their findings also identifying that the desire for anonymity is often hypothetical, as some residents are comfortable with non-anonymous, timely constructive feedback. Anonymity prevents the immediate timeliness of constructive feedback but appears to be a necessary compromise for some residents, to optimize their constructive feedback to preceptors. When feedback is given anonymously the risk of unhelpful or hurtful comments may arise.³⁰ Having agreed upon statements to provide constructive feedback may reduce this risk. Perhaps the most important drawback of maintaining anonymity and delaying feedback is the prevention of a meaningful two-way dialogue between resident and preceptor which we know is an important aspect of assimilating and applying feedback.²⁵ This new process offering the option of adding constructive feedback to FNs is a first step towards the idea of open dialogue between residents and preceptors to improve preceptor feedback skills. The comments box is a way to further promote that dialogue.

Residents also identified the importance of institutional support to foster a culture where preceptors are open to constructive feedback from residents. They also believe that there needs to be transparent program expectations for preceptors to provide regular feedback. Without a culture that supports regular quality feedback, residents feel guilty about asking for preceptor feedback fearing that it will increase preceptor’s workload. Although residents are vital stakeholders in changing assessment culture, the prime responsibility lies with the institution to set assessment expectations, especially given the reality of the power differential between the residents and program.⁴² Both this study and others identify the importance of an explicit institutional expectation for feedback for both learners and preceptors.^27,43

The electronic platform used within our department made it logistically easy to deliver to preceptors two important elements of effective feedback on their feedback: timely and behaviourally-specific information. Our platform was initially created out of necessity to collect and collate the massive amounts of assessment data for our 140 residents. Now, through the TU option and report-generation function, the electronic platform can also be leveraged to provide meaningful feedback to preceptors about their feedback-giving skills. The electronic platform can be easily programmed such that feedback by residents can be either anonymous or non-anonymous.

Conclusion

This study shows that resident-generated, non-anonymous, timely, behaviourally specific, formative, positive feedback for preceptors is possible. This has the potential to improve preceptors’ feedback giving skills as well as promote residents’ feedback-giving behaviour and reinforce helpful feedback giving practices. These habits can highlight the importance of feedback within a teaching environment changing the assessment culture in the program. Ideally this would also be true for constructive feedback, however, at this time residents want an option for anonymity for this. In addition to adding to the literature a way to provide formative feedback to preceptors, it also demonstrates that residents within a large program are willing to provide this to preceptors that they may not have a long-term relationship with.

Practical take away points

1. Residents are willing to identify, in a non-anonymous timely way, preceptor’s feedback that they find helpful for their learning.

2. Residents are discriminating in which feedback they identify as being helpful.

3. Residents want to also be able to give constructive feedback to preceptors to help improve feedback and have identified wording that they would be comfortable using with preceptors to do that.

4. An electronic assessment system can be leveraged to provide holistic (both positive and constructive) feedback to preceptors in a way that addresses residents’ concerns with providing constructive feedback to preceptors.

Future work

Of all FNs, 11% were awarded a TU designation. We do not know what drives residents to award a TU although we do have evidence it is not entirely done as a “reward.” Monitoring the percentage over time and exploring this further will tease out the reasons why residents use this function. Based on this study we have added a second drop-down menu for constructive feedback using the statements endorsed by the resident focus group. We chose a “light-bulb ah-ha” icon to go with this constructive feedback drop-down menu (the idea of a Thumbs Down icon being too negative). This is a new feature of our FNs found alongside the existing TU feature identifying positive feedback skills, with a new option for the feedback (positive or constructive) to be anonymous. Uptake and use of these constructive themes and anonymous option will be evaluated. Monitoring the use of the anonymous feature over time will allow us to see if this need for anonymity declines over time as feedback becomes a more entrenched practice. Knowing that there are a number of factors impacting recipients’ receptivity to feedback and that not all feedback is acted on,⁴³ a critical next piece will be understanding and optimizing the receptivity of, and use by, preceptors of both the positive and constructive feedback they are given by residents to see if it improves their feedback giving skills. It will also be interesting to see if this bidirectional feedback pathway positively impacts the culture of assessment within our department, increasing both our resident’s and preceptor’s feedback seeking and accepting behaviours.

Limitations of this study

This study is confined to a small sample size in a single program in one discipline. It does not explore the perceived value by residents of the quality of verbal feedback they are given, however the value of written feedback, which was explored, is important in reinforcing verbal feedback and documenting evidence of residents’ performance. It does not include the perspective of preceptors, which will be an important aspect to understand if feedback is to be used to change behaviour.

Appendix A

PART A – Interview questions

Section 1: Feedback

1. What makes formative feedback most valuable to you? (Note: Record on whiteboard)

   1. What do you most like about the written comments you receive on FNs? (Note: Record on whiteboard)
   2. What makes feedback less valuable to you? (Note: Record on whiteboard)
      1. What do you least like about the written comments you receive on FNs?) (Note: Record on whiteboard)

2. How could the feedback you receive from FNs be improved? (Note: Record on whiteboard)

3. The DFM assessment system allows you to receive a great deal of feedback.

   1. How do you feel about that?
   2. To what extent do you look forward to receiving feedback?
   3. When do you ask for FN feedback, if ever? (Prompt: What contexts do you ask for feedback?)
   4. In what circumstances does your preceptor initiate the FNs, if at all?

   Section 2: Constructive Feedback

4. What does the term “constructive feedback” mean to you?

5. Are you familiar with the ‘Thumbs Up’ functionality in the FNs?

   Prompt: Have you ever used the TU function on the FNs?

   

   1. If no, Facilitator states: “In the top right-hand corner there is a ‘thumbs up’ icon. If you click on it when you receive a FN that you think was useful to you, you’ll be able to select the reason you believe this to be so.”
   2. If yes: For those of you who are familiar with the ‘TU’ icon:
      1. Is there anything in the TU functionality that you think should be added to improve the feedback you give to preceptors?

6. How comfortable are you giving preceptors constructive feedback about their feedback giving skills?

   1. What are some circumstances that would make you comfortable giving constructive feedback to preceptors, if any? (e.g., anonymity, type of preceptor-resident relationship)
   2. What would be some circumstances where you might not feel comfortable giving constructive feedback (WPBAs) to preceptors?

   Facilitator:

   • I’m now handing out some examples of phrases of constructive feedback for preceptors that might be used in various circumstances if you were:

     • given no written feedback on your FNs,
     • given feedback that was not supportive
     • given generic feedback
     • given feedback that lacked appropriate processes (e.g., was not timely, wasn’t discussed in an appropriate environment, didn’t reflect the verbal discussion).
   • Please take a moment to read them.
   • Feel free to reword, revise, write new comments directly on the handout. This will help us get a sense of what feels okay to you and what does not

7. Would you consider using these phrases?

   1. If so, when?
   2. If not, why?

8. What do you not like about these phrases?

9. What are some other constructive feedback phrases for preceptors that you think would help them better develop their feedback skills and help you become better family physicians?

10. What is your opinion about adding a function to the FNs similar to the ‘Thumbs Up’ function, so that you could provide pre-determined constructive feedback phrases to preceptors about feedback that is not as useful to your learning needs?

11. If you were going to give constructive feedback, what do you think about it should be called?

12. Do you have any other questions to ask or comments you would like to make before ending this session?

PART B – Examples of phrases of constructive feedback for preceptor’s handout

Instructions

These are the elements of the written feedback on FNs that you do not particularly think are useful. I’m handing out some examples of phrases of constructive feedback for preceptors that might be used in various circumstances if you were:

• given no written feedback on your FNs,

• given feedback that was not supportive,

• given generic feedback, and

• given feedback that lacked appropriate processes (e.g., wasn’t timely, wasn’t discussed in an appropriate environment, didn’t reflect the verbal discussion).

Please take a moment to read them. Feel free to reword, revise, write new comments directly on the handout.

a) No information about what or how to improve

I really like getting positive feedback, however, I need to know how I can improve skills.

You have more experience than I do. I would love to hear if you have a better approach to this.

Thanks for giving me feedback. I would really be interested to hear from you about what and how I can improve.

b) No feedback at all

I am trying to work on improving my skills and some written coaching from you on this topic would be valuable to me.

c) Feedback feels to negative

I would find it helpful if you could give me some coaching on how I can improve.

I know I respond best to feedback when you suggestions about how to improve are supportively worded.

d) Feedback is too wordy

I sometimes get confused about what or how to improve. It might be best for me if your feedback to me is shorter.

Preceptor is commenting on performance that was not directly observed.

I would find it very helpful if you were able to watch me next time.

Appendix B

Emergent themes and categories that identify the FN elements deemed useful by family medicine trainees with supporting quotes.

Theme	Category	Sample of Supporting Quotes
1. Changed and improved practice	a) Built on existing knowledge in application to a clinical encounter	“Reinforced the management principles each time I tried to apply them [to] make sure I understood why each decision was made and [I] could build my knowledge.” “Solidif[ied] my knowledge of the procedure, suggestions for how to improve my efficiency.”
	b) Reinforced confidence in existing knowledge improving performance in a clinical encounter	“This specific patient encounter helped me to be more confident with an approach to talking to prenatal patients about pain management options during labour [and] delivery.”
	a) Highlighted application of existing knowledge in a clinical situations (very often this is a nuance that ‘comes with experience’ — preceptor is sharing his/her experience)	“My takeaway from this note was that it really is OK not to cram every issue that comes up into one appointment.” “Give a tip for improving my management of this case so that next time I can go beyond the standard of care.”
2. Imparted new knowledge	a) Observed preceptors to gain new knowledge	“Highlighted a situation which is not one I have encountered that frequently, and the issues that come along with this.” “But what I find very useful [is] listening to his conversation with the patient and gaining insight into how to systematically approach the topic that was individualized to her goals for pain management.”
	b) Direct teaching by preceptors addressing gaps in knowledge	“Addressed areas of the interaction that I was specifically uncertain about (in this case, the necessary investigations) and helped outline an approach in how to address complex, unusual symptoms such as the one that the patient presented with.”
3. Provided motivation to learn	a) Offered support and encouragement	“Provides an appropriate amount of support through encouragement.” “[It] was also very encouraging and supportive.”
	b) Generated specific learning objectives	“I also had very specific learning objectives for reading.”
4. Confirmed skills	a) Confirmed and/or acknowledged learning and/or competence in specific skills	“I found this field note particularly helpful as it reflected back that I did indeed handle a difficult discussion well.” “Acknowledged my attention to non-pharmacologic management…solidified how important it is to provide comprehensive care for my patients.” “Provided a specific example of a skill that we were recently taught and how I was using it well. In the future I will continue to help patients find small attainable goals that they can work towards.”
5. Promoted reflection	a) Reminded to consider unspoken agenda and biases	“Helped me to remember that not only is it a good thing to advocate for your patients, but when needed it is ok to advocate forcefully.” “Helpful reminder to review your diagnosis when the initial treatment does not work.”
	b) Reflected on improving difficult-to master skills	“Reflect on the communication aspect of this particularly difficult encounter, it made me think about the way in which I'd communicated, and how, in future encounters, I can communicate in different ways to ensure patient understanding.”
	c) Timeliness of feedback facilitated reflection as the clinical encounter was easily recalled	“[The preceptor] submitted this field note right after the occurrence so upon looking at my portfolio a few days later I was able reflect on this experience again soon after the event and look back on it while the experience was fresh in my mind.” “Constructive feedback was even more helpful in that Dr. X completed the Field Note immediately after the encounter so that it was fresh in my mind when I read the Field Note.”
	d) Reminded of the more in-depth teaching provided by the preceptor around the clinical case	“The FN serves as a reminder of a more in-depth discussion.” “[The FN feedback] brought to light something that I know I should be doing but did not realize I wasn’t doing. Exploring protective factors not just for suicidal patients but also for patients with violent thoughts is something I will be more conscious of doing next time.”