Feedback and formative assessment in Competency by Design

Abstract

INTRODUCTION

Competency by Design (CBD) is a form of competency-based medical education implemented in Canadian urology programs since 2018. Regular, multimethod assessments and formative feedback via Entrustable Professional Activities (EPAs) are the cornerstones of CBD. Personalized and regular feedback are the top perceived benefits of CBD by both residents and supervisors; however, evidence shows that in practice, constant feedback-seeking is burdensome, and increased quantity of feedback does not equal increased quality. The experience of CBD implementation has not yet been studied in surgical programs. Our aim was to examine how supervisors and residents have experienced the integration of formative assessment and feedback since the implementation of CBD in a surgical training program.

METHODS

Using data from focus groups, a qualitative phenomenological analysis based on the experiences of the residents and supervisors in a urology residency program was performed.

RESULTS

Residents and supervisors felt that CBD allowed for better tracking of resident performance and increased quantity of feedback; however, increased workload, delayed completion of EPA assessments, lack of direct observation in non-surgical activities, variable supervisor guidance, and lack of understanding of CBD were cited as barriers to providing proper feedback and formative assessment.

CONCLUSIONS

The participants experienced a lukewarm transition in feedback and formative assessment practices with CBD. As with every process of change, these growing pains may eventually result in meaningful practice improvements and incorporation of a CBD culture into everyday learning activities.

INTRODUCTION

Competency by Design (CBD) is a form of competency-based medical education (CBME ) for resident training that was implemented into urology training programs in 2018.^1,2 The goals of CBD are to create a resident-centered learning experience and an outcomes-based approach, which allows educators to evaluate resident performance based on predetermined tasks and criteria.^3,4 In CBD, these predetermined criteria are called milestones and Entrusted Professional Activities (EPAs), and are based on the CanMEDS competencies.^4–6 Frequent, multimethod assessments and formative feedback are the cornerstones of CBME.^7,8 Feedback and formative assessment are intimately linked, as assessment evaluates learner competence and provides meaningful feedback to drive learning.⁹

Feedback should include all three elements of “Feed up, feed back, and feed forward” to be effective, meaning that 1) feedback must be goal-oriented and the end goals must be clear; 2) feedback on the process towards attaining the goal must be given; and 3) the next activities to be undertaken by the learner to ensure progress should be clear. Four different levels of feedback exist: 1) task feedback — how well the task is understood or performed; 2) task processing feedback — main processes needed to complete the task; 3) self-regulation feedback — learner self-monitoring of their own progress; and 4) self-as-a-person feedback — personal, affective evaluations about the learner.¹⁰ Formative assessment (assessment for learning) differs from summative assessment (assessment of learning), as the former provides continuous assessment, which in turn, drives continuous learning (Table 1). ^11–13

Table 1.

5 key strategies of formative assessment per Wiliam & Thompson

	Strategy	Description
1	Clarifying and sharing intended learning objectives and criteria for success.	The teacher must ensure that the learning objectives and expected outcomes of the curriculum are clear for the students.
2	Engineering effective discussions and learning opportunities/learning tasks that elicit the student’s understanding and competence.	The teacher should facilitate learning experiences and opportunities for the students that reveal their current level of competence.
3	Providing feedback that moves the student forward.	The teacher should provide feedback to their students in a “feed up, feed back, and feed forward” fashion (Hattie & Timperley, 2007) to facilitate student progression through the curriculum.
4	Activating peers as instructional resources for one another.	Sharing knowledge and learning experiences with peers may enhance the student’s learning experience.
5	Activating the student as owner of their own learning.	The student must grasp the learning objectives and outcomes that are expected of them and implicate themselves as responsible for their own learning.

Wiliam & Thompson, 2007

More personalized and regular feedback has been identified as one of the top perceived benefits of CBD by both residents and supervisors in Canadian urology training programs;¹⁴ however, although residents in a CBD-based internal medicine program valued the idea of feedback, in practice, constant feedback-seeking was burdensome.¹⁵ The quantity of feedback had increased but the quality had decreased, and constant assessments interrupted daily workflow and learning opportunities.¹⁵ A survey conducted on residents in their third year of CBD in the province of Quebec demonstrated that residents rate their level of satisfaction with feedback received via EPAs as only 5/10.¹⁶ Although EPA-based assessments provide standardized outcomes for residents, they have been shown to decrease opportunities for individual learning experiences.¹⁷ Other challenges, including inconsistent faculty engagement and lack of direct observation, have also been described as barriers to formative assessment in CBME.¹⁸ Furthermore, medical and surgical training programs differ greatly.

To our knowledge, no study to date has explored the experience and perceptions with feedback and formative assessment within CBD for both residents and supervisors specifically within a surgical training program, such as urology. Our aim was to examine how supervisors and residents have experienced the integration of formative assessment and feedback since the implementation of CBD in a surgical training program.

METHODS

Setting & participants

This study was performed within an accredited, five-year Canadian urology residency program. Participants included the nine junior residents in the CBD cohort at the time of data collection (residency training years 1, 2, and 3) and 20 urology supervisors.

Design & methodology

A phenomenological, qualitative approach was selected to explore and understand the experiences of the urology residents and supervisors by studying their opinions and perspectives.¹⁹ A total population sampling method was used.²⁰

Two focus groups were held: one for the CBD residents, and another for the supervisors (Appendix available at cuaj.ca). The focus groups were approximately 90 minutes in duration and were led by the lead researcher of this study, a urology supervisor with Master’s training in health profession education. An independent research assistant was present to monitor and ensure objectivity of the interactions.^21,22 The focus group guides were pilot-tested to ensure the questions would provide meaningful data.²³ The focus groups were audio-recorded then transcribed verbatim. All data were de-identified during the transcription process.

Analysis

The narrative data produced by the transcription of the recorded data were analyzed by two researchers. The seven sequential steps of Colaizzi’s descriptive phenomenological method were used to structure data analysis.²⁴ Thematic analysis was used to formulate meaning and code significant statements.²⁵ The two researchers independently coded part of the transcripts, then met to compare codes and discuss. Each significant statement and identified codes were reviewed as part of peer debriefing. In cases of coding inconsistencies, a discussion occurred, and consensus was obtained. Both researchers met again after analysis of the remaining data to compare and discuss meaning, codes, and initial themes. Bracketing of presuppositions was essential during this process to remain as close as possible to the phenomenon as described by the participants.²⁴ Data was then re-examined using insight gained by analysis, and meanings were clustered into themes.

This study was approved by the local institutional ethical review board (study number A01-809-218).

RESULTS

Five of the nine CBD cohort residents and 13 of the 20 supervisors were present for their respective focus groups. Five major themes, several subthemes, and a domain summary encompassing the global experience of all participants²⁶ were identified (Table 2). No new themes were identified by the end of the data analysis of both focus groups, therefore, no other focus groups were held, as the researchers felt that data saturation was achieved. Quotations are identified with group (R=resident vs. S=supervisor) and participant (P) number.

Table 2.

Identified themes, subthemes, and significant participant statements

	Theme	Subthemes	Significant participant statements
1	Experience of feedback reception and delivery	– Verbal vs. written (EPA) feedback – Exchanging feedback	“EPAs are more just a sentence, a one-liner, or even a word or two. It’s very short in comparison to verbal feedback.” (RP3)
2	Impact of feedback and EPA assessments of achieving training objectives	– Learning objectives – Facilitators and barriers to feedback	“When the residents think that they’ve done an excellent job, that’s when they send the EPAs and that’s why we get a skewed view of their performance. They don’t send them when they’re struggling or for things that they need to improve.” (SP8)
3	Attitudes and perceptions in relation to feedback and formative assessment		“EPAs are useful tools to advocate for your learning experience and overall directs your learning. But it creates anxiety to count everything you do instead of reading up on topics that you’re working on.” (RP2)
4	Impact of time and timing on feedback and formative assessment	– Time to feedback provision – Frequency of feedback – Time as a resource	“We give a lot more feedback to the residents. Before, it was mid-rotation, end of the rotation, you would sit down, but now it’s a daily process. So, I personally give a lot more feedback.” (SP6)
5	Impact of location on feedback and formative assessment		“We follow up on the patient consultation, but I don’t go to the emergency room to look at the resident do an assessment, for example. Yet, I fill these EPAs if they seem to be complete and make sense.” (SP6)

EPA: Entrustable Professional Acitivity; P: participant; R: resident; S: supervisor.

Domain summary: Global participant experience

Globally, both positive and negative experiences regarding EPA-based feedback and formative assessment in CBD were described by the participants. Both groups expressed that CBD was helpful to track residents’ training goal achievement and monitor their progress. Continuous feedback and assessments decreased the perceived high stakes of summative mid- and end-of-rotation feedback.

“There are so many [evaluations that] it removes the pressure of the final evaluation.” (RP5).

The residents described that CBD allowed them to ensure adequate and equal exposure to learning activities throughout their training program. The supervisors expressed that residents were increasingly autonomous in seeking feedback since CBD implementation; however, certain aspects of feedback and EPA-based assessments were negatively perceived by the residents. For example, constant “counting” and tallying of EPA assessments and feedback-seeking were described as time-consuming and stressful. One resident deplored this as being a barrier to learning goal attainment, as time was spent requesting and tallying EPA assessments rather than studying.

“[CBD is] too much administrative work, too much counting o f everything. It takes a lot from my own learning b ecause I’m not spending those hours reading, I’m spending t hem trying to figure out which EPAs [I need].” (RP2)

In some circumstances, residents would prioritize a less stimulating learning experience if an important EPA that they required was associated with it. The residents were disappointed that not all supervisors filled out the EPA assessments in a timely fashion, placing a large burden of responsibility on them to provide reminders.

“[CBD] takes away from your learning because it’s very t ime-consuming to run after the staff to have them fill your EPA and to deal with the frustration of sending an EPA a nd never having it filled out.” (RP2)

Furthermore, the residents described that if their EPA feedback was filled in a delayed fashion by a supervisor (more than one day after the learning activity), this resulted in more generic and less useful feedback.

The supervisors echoed many of the negative sentiments expressed by the residents. They denounced that the software for CBD is not user-friendly, nor convenient, and written feedback is time-consuming. Two supervisors expressed that they “don’t like” CBD and “[were] not convinced” that CBD improved resident assessment and performance compared to the traditional pre-CBD curriculum.

Theme 1: Experience of feedback reception and delivery

Verbal vs. written (EPA) feedback

Feedback described by the participants was globally separated into “verbal” feedback and “written/EPA-based” feedback. Verbal feedback tended to be delivered immediately or at the end of the task/day, was more often personalized and task-directed, and was described as of higher quality and superior to written feedback, especially from the supervisors’ points of view. Task and task-processing feedback were perceived by both the residents and supervisors as being the most used type of feedback given in ample amounts, especially in a verbal fashion, during surgical procedures.

Written feedback tended to be delivered in a delayed fashion, on average a few days after the learning activity but on occasion up to three weeks later. Written feedback was described to often be of a self-as-a-person level, which was perceived as “generic” by the residents.

“EPAs are more just a sentence, a one-liner, or even a w ord or two. It’s very short in comparison to verbal feedback.” (RP3)

“The longer it takes for the EPA to be filled, the more it’s g eneric and less constructive because staff forget specifically w hich steps we should have improved and which s teps we did better.” (RP4)

“Verbal feedback carries more weight for a supervisor than s omething written down the night or the day after. Often, when it comes to filling out the EPA, you forgot the small t hings that bothered you or that you felt they need to w ork on, and you just focus on the bigger objective.” (SP7)

Exchanging feedback

The concept of feedback-seeking (by residents) was described as onerous and time-consuming.

“I find that it generates a lot of stress for me. I find that it’s v ery time-consuming to count how many EPAs are done, which EPA is missing or not, [etc.].” (RP2)

The residents sometimes had to remind the supervisor on multiple occasions to fill out EPA-based assessments. On occasion, EPA evaluations expired before the supervisor had completed it, which created a negative experience for the resident due to lost opportunities for feedback and completion of learning objectives.

Theme 2: Impact of feedback and EPA assessments on achieving training objectives

Learning objectives

All participants expressed that CBD helped provide clear training objectives for the residents, as the list of EPAs to be achieved presents itself like a list of learning objectives to acquire throughout residency; however, in practice, the residents expressed that specific expected training objectives were not usually reviewed with them at the beginning of each rotation.

Many supervisors did not explicitly review which training objectives were relevant for their site at the beginning of the rotation, and they expected this task to be the responsibility of the program or site director.

Facilitators and barriers to feedback

Some supervisors expressed that residents were hesitant to request EPAs if they did not perform “well” on an activity, and that this was a barrier to providing feedback because they were limited in providing assessments on tasks less well-performed.

“When the residents think that they’ve done an excellent j ob, that’s when they send the EPAs and that’s why we g et a skewed view of their performance. They don’t send t hem when they’re struggling or for things that they need t o improve.” (SP8)

A “CBD culture” was described as both a facilitator and a barrier to feedback. Current attitudes — for example, supervisors stating that they “do not believe” in CBD — were described as barriers to providing and receiving feedback through EPAs. The residents expressed that it was difficult to request EPA evaluations from supervisors who were less open to the concept of CBD, and some supervisors reportedly refused to fill EPA-based assessments; however, some participants felt that CBD would become easier with time.

“It’ll be interesting to see people that were part of the CBD cohort when they’re the ones that are becoming staff a nd are evaluating other people. I feel that over time, the i mportance of the EPAs will be better integrated into the u sual routine.” (RP5)

Theme 3: Attitudes and perceptions in relation to feedback and formative assessment

The residents described themselves as autonomous and as self-directed learners, and that they were responsible for ensuring they completed all of the required EPAs; however, this level of responsibility was also felt to be a burden on top of an already stressful surgical residency.

“EPAs are useful tools to advocate for your learning experience a nd overall directs your learning. But it creates anxiety t o count everything you do instead of reading up on topics t hat you’re working on.” (RP2)

Theme 4: Impact of time and timing on feedback and formative assessment

Time to feedback provision

Residents and supervisors described three moments at which feedback is usually provided: 1) immediate/in-the-moment feedback; 2) end-of-the-day/end-of-the-activity feedback; and 3) delayed feedback (several days after the activity was performed).

Immediate/in-the-moment feedback was exclusively described as being given verbally, most often during surgical procedures. Task and task-processing feedback were the most common levels of feedback that were immediately delivered, most commonly during surgical procedures. The residents felt that immediate/in-the-moment feedback was very frequently and explicitly given and was useful to help them progress through a task and achieve their learning goals.

“It’s important to let us know in real-time how we’re doing, like with the [task] level of feedback.” (RP4)

The participants expressed that end-of-the-day/end-of-the-activity feedback was less frequently given and most often self-regulatory in nature. Negative sentiments surrounding delayed feedback were very commonly expressed. This issue related exclusively to written EPA feedback. Delayed feedback was due to delays in submitting EPA requests by the residents and delays in completion by the supervisors.

Frequency of feedback

Since CBD was implemented, feedback was described as being given more regularly by the supervisors, both verbally and written via EPAs.

“We give a lot more feedback to the residents. Before, it w as mid-rotation, end of the rotation, you would sit down, but now it’s a daily process. So, I personally give a lot more f eedback.” (SP6)

The increased frequency of feedback was seen as positive by the residents, as it decreased pressure from the end-of-rotation evaluations and helped them monitor their own progress.

Time as a resource

Lack of time as a resource was very frequently mentioned and was perceived as a major barrier to feedback provision. Time restraints were felt to cause delays and subsequent decrease in quality of written EPA feedback. The participants expressed that the new CBD curriculum increased their already heavy workload.

Theme 5: Impact of location on feedback and formative assessment

Participants felt that non-surgical activities received much less direct observation, which limited the quality of feedback in these settings, as feedback tended to rely on the resident’s reporting skills rather than their true performance.

“We follow up on the patient consultation, but I don’t g o to the emergency room to look at the resident do an a ssessment, for example. Yet, I fill these EPAs if they seem t o be complete and make sense.” (SP6)

DISCUSSION

EPA-based assessments in CBD serve two purposes: 1) monitoring resident competence and achievement of learning goals through formative assessment; and 2) providing continuous and meaningful feedback to residents to guide their learning. The integration of CBD into our urology program has sparked various reactions from the participants, as they expressed their personal and group experiences with this curricular change.

The participants appreciated that EPA assessments allowed them to better track resident progress. Despite this scaffold created by CBD, the residents expressed that explicit learning objectives were not regularly reviewed with them, and some supervisors did not believe that this was their responsibility. Nonetheless, clarifying intended learning objectives and criteria for success is one of the key strategies in formative assessment.¹³ Inconsistent faculty engagement has been described as a barrier to formative assessment in CBME.¹⁸ Furthermore, per the feed up, feed back, and feed forward” concept, feedback must be goal-oriented to be effective.¹⁰

The residents reported that constant tallying of EPA assessments and feedback-seeking were time-consuming, stressful, and a barrier to learning goal attainment; however, activating residents as owners of their own learning is one of the key strategies of formative assessment.¹³ Despite this, it appears that residents feel constrained to “fit the mold” of CBD rather then to explore unique or meaningful learning experiences. Effectively, Martin and colleagues demonstrated that although EPA-based assessments provide standardized outcomes for residents, this in turn decreases opportunities for individual learning experiences and objectives,¹⁷ which does not align with resident-centered learning.^4,5

Participants described a lack of direct observation during non-surgical activities, and this was perceived as a barrier to proper assessment and feedback. This lack of direct observation was also reported by Upadhyaya and colleagues as a barrier to implementation of formative assessment in CBME.¹⁸

Furthermore, supervisors expressed that residents would seek EPA evaluations for activities they knew they would “pass” rather than continuously requesting EPA assessment throughout the process of their learning curve. This loss of the meaning behind the EPA translates to a skewed perception of resident performance, and in turn causes EPAs to be used as summative assessments rather than formative, causing a “chicken-or-egg” type of scenario. Per the “feed up, feed back, feed forward” notion, feedback on the process towards attaining the learning goal must be given to be effective.¹⁰ Lack of time as a resource for both supervisors and residents may also contribute to this trend.

The implementation of CBD introduced a change in feedback practices, including written and more regular feedback. Effectively, an increase in personalized and regular feedback was identified as one of the top perceived benefits of CBD in Canadian urology training programs;¹⁴ however, written feedback was often delayed and generic. The quality of feedback received through EPA-based assessments is a known issue in the province of Quebec.¹⁶ Nonetheless, optimistic outlooks on the future of CBD, feedback, and assessment practices in the program were expressed. The Royal College and the Fédération des Médecins Résidents du Québec recently emitted a recommendations report highlighting many similar experiences to those described in this study and recommended actions for change to help programs, supervisors, and residents navigate this culture shift.¹⁶

Limitations

This study was not without its limitations.

This study was performed during the initial stages of CBD implementation and participants’ experiences may have already changed. The experience of CBD in senior residents may differ from that of their younger colleagues, as residents increasingly gain confidence and competence.

Furthermore, in phenomenology, the researcher delves into the lived experiences, opinions, and feelings of the participants. The principal researcher in this study is a supervisor in this program and possesses their own lived experiences and opinions that could impact data collection and analysis; however, the use of reflexivity strategies, such as bracketing²² and a research assistant for peer debriefing, mitigated this impact.

The use of focus groups in a phenomenological approach may be considered a limitation; however, focus groups have been shown to be compatible and even beneficial to phenomenological research, as focus groups stimulate group discussion and allow the researchers to explore a variety of perspectives.²⁷

Finally, the generalizability of this study’s results are limited by the fact that it is a small, single-institution study.

CONCLUSIONS

The participants in our program experienced a lukewarm transition in feedback and formative assessment practices since CBD was implemented in 2018. They expressed that CBD allows for better tracking of resident performance and increased quantity of feedback; however, increased workload, delayed completion of EPA-based assessments, lack of direct observation in non-surgical activities, variable supervisor guidance and involvement, and lack of understanding of CBD were felt to be barriers to providing proper feedback and formative assessment. As with every process of change, these growing pains will hopefully eventually result in meaningful practice improvements and incorporation of a CBD culture into everyday learning activities. Strategies developed by the Royal College and Canadian medical societies exist to help supervisors, residents, and residency programs tackle negative experiences and navigate this culture shift.

KEY MESSAGES.

• ■ CBD is a form of medical resident training that combines an outcomes-based approach and a resident-centered learning experience.

• ■ Frequent, multimethod assessments and formative feedback are the cornerstones of CBD.

• ■ Since CBD was implemented into the curriculum of a Canadian urology training program, both residents and supervisors felt that it improved tracking of resident performance and increased quantity of feedback.

• ■ Certain barriers to proper feedback and formative assessment have been identified: increased workload, delayed feedback, lack of direct observation, variable supervisor engagement, and lack of understanding of CBD.