“The Most Crushing Thing”: Understanding Resident Assessment Burden in a Competency-Based Curriculum

Abstract

Background

Competency-based medical education (CBME) was expected to increase the workload of assessment for graduate training programs to support the development of competence. Learning conditions were anticipated to improve through the provision of tailored learning experiences and more frequent, low-stakes assessments. Canada has adopted an approach to CBME called Competence by Design (CBD). However, in the process of implementation, learner anxiety and assessment burden have increased unexpectedly. To mitigate this unintended consequence, we need a stronger understanding of how resident assessment burdens emerge and function.

Objective

This study investigates contextual factors leading to assessment burden on residents within the framework of CBD.

Methods

Residents were interviewed about their experiences of assessment using constructivist grounded theory. Participants (n=21) were a purposive sample from operative and perioperative training programs, recruited from 6 Canadian medical schools between 2019 and 2020. Self-determination theory was used as a sensitizing concept to categorize findings on types of assessment burden.

Results

Nine assessment burdens were identified and organized by threats to psychological needs for autonomy, relatedness, and competence. Burdens included: missed opportunities for self-regulated learning, lack of situational control, comparative assessment, lack of trust, constraints on time and resources, disconnects between teachers and learners, lack of clarity, unrealistic expectations, and limitations of assessment forms for providing meaningful feedback.

Conclusions

This study contributes a contextual understanding of how assessment burdens emerged as unmet psychological needs for autonomy, relatedness, and competence, with unintended consequences for learner well-being and intrinsic motivation.

Objectives

Investigation of contextual factors leading to resident assessment burden in Competence by Design.

Findings

In operative/perioperative training contexts, entrustable professional activity assessments were experienced as sources of stress and added workload for learners that were paradoxically high stakes and low value. These assessment burdens emerged as unmet needs for autonomy, relatedness, and competence.

Limitations

Due to the qualitative design and contextual nature of this study, findings are not generalizable to all training programs. Insights will be most relevant for similar competency-based programs.

Bottom Line

Self-determination theory offers strategies to mitigate assessment burden and improve learner well-being and competence in competency-based training.

Introduction

Faculty assessment burden was expected to increase as a consequence of implementing competency-based medical education (CBME). The additional workload directed to formative assessment is part of a paradigm shift in teaching and assessment practices in the CBME era.^1-4 However, the conceptual literature on CBME did not anticipate assessment burdens residents may experience as training programs transition to CBME. This may be a problem, because CBME promises to produce more competent physicians and surgeons through improved learning experiences.^1-6 Learning is often difficult work. However, increased workload can be a factor in physician burnout and ill health.^7-12 If residents are experiencing assessment burdens in CBME, there may be unintended consequences. Wellness and competence could be at stake.

To mitigate unintended consequences, we need contextual understandings of how assessment burdens emerge. There are different approaches to CBME.⁴ In the Canadian context, one such model is Competence by Design (CBD), developed by the Royal College of Physicians and Surgeons of Canada.¹³ Findings from implementation research suggest residents in CBD training programs are experiencing assessment burdens that impact their well-being.^14-19 In these reports, residents describe stress and anxiety related to the work of acquiring assessments of entrustable professional activities (EPAs). Missing from discussion of these problems, however, is an understanding of the conditions through which these burdens emerge and function. This grounded theory study contributes to contextual research on this question.

Methods

Constructivist grounded theory (CGT) is an ideal methodology to develop explanatory models of how effects emerge in practice for 2 reasons. First, CGT uses in-depth, semi-structured interviews to investigate participant experiences, generating theory richly grounded in contextual data.^20,21 Second, CGT is an emergent methodology, affording exploration of questions that arise in the research process through an iterative approach to data collection and analysis.^20,21

Setting and Participants

The study is part of a research program investigating how CBD translates to practice in operative (surgical) and perioperative (anesthesia) training programs. More information on CBD as an approach to CBME, how EPAs are developed and assessed, and how competence committees function can be found on the Royal College website.¹³ CBD is being phased into Royal College accredited training programs between 2017 and 2022. Data collection for this study occurred between 2019 and 2020; therefore, our sample includes only training programs that had begun implementation by 2019.

Initial data collection was conducted in the departments of Surgery and Anesthesia & Perioperative Medicine at Western University in London, Ontario, Canada. Residents in operative and perioperative training programs in years 1 to 3 of implementation of CBD were invited to participate. Consenting residents (n=11) were in a range of training phases and came from surgical foundations, urology, general surgery, obstetrics and gynecology, and anesthesiology programs. We do not provide more specifics to protect the identities of the participants, given that use of direct quotations is potentially identifiable if linked with other data.

Interviews averaged 40 minutes and asked residents to describe their experience with EPA assessment. The interview guide is available as online supplementary data. Initial coding of the interview transcripts established a typology of assessment burdens. We questioned whether these were due to systemic issues with CBD or factors specific to the local context. To explore this question further, we invited Canadian residents in similar training programs to participate. Recruitment was conducted via Twitter using a link to the study letter of information and consent in the post.

In the phase of data collection from external contexts, residents (n=10) also represented surgical foundations, urology, general surgery, obstetrics and gynecology, and anesthesiology programs. In the representation of results, we use the following nomenclature to distinguish quotes from the local and external samples. For example, LC001 indicates a quote from participant 1 in the “local context,” while EC102 indicates participant 12 from an “external context.” This decision reflects our intention to represent systemic patterns in the data while protecting confidentiality. The combined data set is a purposive sample of 21 residents, represented in the Table. The sample size was determined by conceptual depth criteria²²; we required multiple sources of evidence from both contexts to corroborate each finding and aimed for matched samples in numbers and types of training programs. The number of participants was also aligned with typical sample size justifications for qualitative research.^23,24

Table.

Study Contexts

Resident Sample	Institutions	Programs Sampled
Local context (n=11)	1. Ontario Medical School	Surgical foundations, urology, general surgery, obstetrics and gynecology, and anesthesia
External context (n=10)	3. Central Canada Medical Schools 2. Western Canada Medical Schools

Data Analysis

One investigator (R.P.) conducted all interviews and did initial line-by-line coding to identify a typology of assessment burdens, meeting regularly with M.C.O. and S.C. to share emerging insights. When preliminary findings were shared with the entire research team, the decision was made to extend the study to residents from external contexts. Another investigator (M.C.O.) recoded the entire data set and found consistency in burdens across contexts through the process of constant comparison.^20,21 Both rounds of coding also identified a prevalent theme in which participants associated their experiences of EPA assessment with feelings of anxiety or stress. As we sought to explain how assessment burdens emerged as stressors in resident experiences of CBD, we found that self-determination theory (SDT) offered the best conceptual model. The justification for keeping a sensitizing concept within a grounded theory study is whether it offers the best explanation for the data at the end of the analysis.²⁰ SDT offers a comprehensive theory of the conditions that support or detract from learning and well-being,^25-27 with promising applications to medical education.^28-31 SDT stipulates that autonomy, competence, and relatedness are interdependent psychological needs, the constituents of intrinsic motivation, development, and well-being. Performance and well-being have been shown to improve when these needs are met and to suffer in their absence.^25-27 Therefore, unmet needs for autonomy, competence, and relatedness are potential sources of “ill-being.”²⁵ In a final round of coding, M.C.O. analyzed each type of assessment burden through the lens of SDT to identify if and how the burdens represented unmet needs for competence, autonomy, or relatedness.

Reflexivity

Transparency is essential for reflexivity in research.^20,21,24 While serving as a surgical program director, the senior author conceptualized the study to identify and mitigate unintended consequences of CBD in this context. Other collaborators contributed to recruitment and confirmation of findings through their experiences of implementing CBD in their surgical and anesthesia training programs (M.C., J.V.K.). Members of the research team with expertise in qualitative research in education conducted the interviewing and preliminary data analysis (R.P., M.C.O., S.C.). The first author has used SDT as a framework in other work exploring well-being in other contexts.^32,33 Our backgrounds have inherent affordances and limitations for our ways of seeing.²⁴ Meeting as a team with diverse perspectives on education allowed us to remain open to different interpretations. The study was approved by the Research Ethics Board at Western University.

Results

Residents carried the weight of 9 assessment burdens associated with CBD: (1) missed opportunities to self-regulate learning; (2) lack of situational control; (3) comparative versus formative assessment; (4) lack of trust; (5) time; (6) connection; (7) clarity; (8) unrealistic expectations; and (9) limitations of assessment forms to provide meaningful feedback. Beyond the strains of additional workload, when considered through the lens of SDT, these burdens functioned as threats to meeting needs for autonomy, relatedness, and competence. Using SDT as an organizing framework, Figures 1 to 3 represent the burdens in categorical relationship to these unmet psychological needs, with representative quotes. The following text explores the properties of each need and the effects of the assessment burdens in more detail. While the assessment burdens have distinct features, they are related in a way that had a compounding impact. Assessment burdens resulting from unmet needs in one area had ripple effects on other needs, causing multiple sources of stress and decreased motivation to engage in EPA assessment.

Figure 1.

Threats to Autonomy Figure

Figure 3.

Threats to Competence

Autonomy

In the category of threats to autonomous motivation, externally imposed limitations on a resident's ability to pursue meaningful learning opportunities caused frustration and depleted autonomous motivation to develop mastery. This assessment burden was experienced as a missed opportunity to self-regulate one's learning: “I'm definitely not feeling more control or being in charge of my learning at all” (EC108). A lack of situational control added further stress when individuals felt held accountable for achieving EPAs (such as putting in chest tubes) when opportunities did not present themselves. Residents also described feelings of worry concerning the high stakes of garnering enough successful EPA observations compared to others in their program: “I'm stressed always to make sure I get these done, and that I'm keeping up with the rest of the cohort as well, and not falling behind” (LC006). At the same time, residents universally expressed that documented feedback on the assessment observations was of little value to their formative development: “I'm like worried about these evals that I don't think are all that helpful” (LC003). This paradoxically high-stakes/low-value experience reduced intrinsic motivation to an extent described by one resident as, “It feels like you're participating in a really dumb game” (LC001).

Relatedness

Relatedness in SDT refers to the need to feel belonging and care in one's social systems. In the training programs we sampled, relatedness was threatened by 3 assessment burdens representing unmet needs for trust, time, and connection in relationships with clinical supervisors. A lack of trust developed when residents felt unfairly judged by the competence committee, as the quote in Figure 2 demonstrates. Lack of time and energy was cited as a burden residents experienced in the quest for numerous EPA observations that placed a strain on their relationships with clinical teachers. The quote in Figure 2 from LC002 refers to the discomfort of having to ask faculty for multiple assessments to complete requirements in the compressed time frame of a training rotation. Requests for EPAs also sapped energy from residents and teachers alike. As one resident described it:

Figure 2.

Threats to Relatedness

“I've had days where I think this is excellent, this is perfect for an EPA, but what ends up happening is 2 more consults come in or something and we get distracted, and by the end of day everybody's exhausted and you just don't even want to bother” (EC109).

A lack of connection was also described as a source of stress, in which residents noted instances when support from faculty and other workplace supervisors to complete EPAs was absent. The surgical foundations EPAs were noted to be especially problematic. For example, the quote in Figure 2 about difficulty getting feedback from “the multidisciplinary team” (LC001) refers to surgical foundations, where residents are working in training blocks that are not necessarily in their home programs. As another resident discussed, “I felt that for the earlier stages, especially for the surgical foundations… there were some EPAs that were a bit difficult to achieve if you were not on a particular service. So those were a little bit more stressful to get” (EC106).

Competence

Competence improves when learners are supported by clear and realistic expectations and task-related feedback. In this study, residents described a lack of clarity in the way EPA criteria were communicated, both directly: “They send out an email after they review you at competency and then they tell you what EPAs you've achieved and then that's it…. Well, that's not very helpful” (EC102); and indirectly: “It's hard to really know which one you're missing…all those [contextual variables] aren't on the spreadsheet” (EC107). Another burden was experienced as unrealistic expectations, as the quote in Figure 3 about intubations being “hard to get because it is more of a senior task” (EC108) demonstrates. The overwhelming experience in operative and perioperative training, however, was that in-the-moment feedback was formative, but that assessment forms “didn't provide any direction” (LC007) because they lacked the tactile specificity of this learning interaction:

“For example, let's say we're in the operating room and I'm doing the cholecystectomy and then the staff is saying you need to, for example, work more with your left hand, kind of pay more attention to the control of the tip of your instrument…. Whereas if you get the feedback which is the electronic form, I find that it may be more general unless the staff really remembers certain points of what you were doing” (EC106).

In the operative/perioperative context, therefore, feedback on assessment forms was experienced as an uninformative evaluation: “I don't think that looking at these tick boxes is helpful, it's not an interaction…I think it's more of a, please get this done so I can be progressed to the next level” (EC109). Residents felt the burden of time to complete the “tick box” form was of little value because it did nothing to further their competence beyond the feedback received during the procedure itself.

Compounding Burdens

Competence, relatedness, and autonomy are interdependent needs according to SDT. Examples in our data illustrated how an unmet need in one area could have compounding effects on other needs, creating multiple assessment burdens. Take, for example, the resident who expressed: “It's hard to know which one [contextual variable of an EPA] you're missing” (EC107). The lack of clarity meant this resident did not have the data they needed to master the competency. But this also reflected an unmet need for support from their teachers to understand the criteria and a missed opportunity to self-regulate their learning.

In the case of the resident who said, “I can't create situations necessarily, and I think the one thing that I've struggled to get…was an intubation. Intubations are hard to get because it is more of a senior task” (EC108), unrealistic expectations and a lack of situational control created this struggle. This experience of unproductive struggle had a negative impact on their intrinsic motivation, sense of self-efficacy, and feeling of support from their training program.

Likewise, the resident who expressed that it felt like they were being driven to do something “that is only serving the means of the Surgical Foundations Competence Committee and doesn't seem important to you at all” (LC101) experienced an assessment burden that threatened both their autonomy and sense of relatedness to their program.

Finally, we learn more from residents in our study about the compounding stress of asking for feedback all the time:

“Just imagine going to your current boss and they're going to tell you that every day you need to send them a form because they're going to try to identify your weaknesses and your strengths and target them. Then, they're going to expect you to send a form every day and you'll be evaluated by a committee based on that… I guess, we're performing at a certain level and we're all pretty Type A people, so I think it can become kind of stressful.” (LC001)

Not only was this stress associated with relational discomfort: “It's uncomfortable to ask people for feedback all the time,” it was a burden that drained time and energy: “and your phone battery's always low and they're always running around” (EC102). However, “the most crushing thing [was] when they say yes…they make it easy for you to get it filled out but then you don't get any real feedback” (EC102). In sum, residents found that the burden of continuously asking for EPA assessments had a compounding impact on both their well-being and intrinsic motivation to engage in the process as a source of stress that did not provide meaningful feedback.

Discussion

In this study's context of CBD, we identified 9 assessment burdens residents experience in operative and perioperative training programs. These burdens encapsulate resident experiences of lack of control, stress, and unproductive struggles and represent unmet needs for autonomy, relatedness, and competence as defined by SDT.

SDT offers an explanation for how assessment burdens in this context emerged as unmet psychological needs and how they may function as threats to competence and well-being. For example, autonomy, understood as intrinsic motivation in SDT, is associated with the development of self-regulated learning.^34,35 Research in SDT has found that assessments evaluating performance relative to others impede intrinsic motivation to set learning goals that lead to mastery.²⁷ In our study, there were cases in which residents expressed stress due to perceptions of unfair comparisons by the competence committee. This assessment burden emerges as an unmet need for understanding and support from their training program, results in relational stress, and may affect intrinsic motivation to engage in self-regulated learning.

The design of our study and approach to sampling emphasizes the consistent and cross-cutting nature of assessment burdens in CBD in operative and perioperative contexts. Our results demonstrating the high stress of managing EPA acquisition compared to the low value of the experience for learning resonates with research on resident experiences in other training contexts that have implemented CBD.^14–16,18 Together, these results show that the promise of CBME to provide low-stakes, high-value formative assessment leading to improved learning experiences and enhanced well-being for residents has not yet been realized in implementation of CBD. Our study offers insights to mitigate this unintended consequence.

The findings on assessment burdens that relate to lack of connection and clarity in how programs communicate about EPAs with residents, problems with completion of assessment forms, and unrealistic expectations, highlight the importance of teachers and learners having a shared understanding of EPA criteria, technologies to support real-time feedback, and quality improvement of EPA curricula. Others are making similar arguments that focus on a principled and contextual approach to implementation of CBD.^6,17,36 What we have shown in our study is that strategies to support needs for autonomy, relatedness, and competence must be addressed holistically, at both local and national levels of CBD implementation.

The key is to focus on the interdependence of learning conditions. According to SDT, the following learning conditions provide essential supports for relatedness, competence, and autonomy: understanding and care, clear and realistic expectations with task-related feedback, and opportunities for choice in learning goals.²⁷ When choice is not an option, the value of the required goal must be internalized by the learner so that it can become an intrinsic goal.^25,27 The good news is these learning supports are familiar strategies for medical educators.³¹ In fact, they are already components of the theory of CBME.^1,4,35

Supporting Relatedness

For example, various forms of coaching are receiving attention for their applications to CBME.^19,37-40 Coaching meets the relational need for learners to feel supported by their teachers and fosters the development of competence.¹⁹ Furthermore, some forms of coaching provide autonomy supports by helping learners set mastery goals.^38-40 To be maximally effective as a strategy for helping residents thrive in CBD, however, faculty development must be reoriented with the understanding that assessment in CBME requires a coaching mindset of all teachers.

Supporting Competence

Programmatic assessment in CBME includes an approach described as assessment for learning.³ Assessment for learning is a comprehensive approach to teaching that includes formative assessment, but “feedback” is only part of the equation.^41-43 When assessment for learning functions as intended, teachers are the originators and first users of assessment data, analyzing it to plan next steps.⁴¹ Then teachers coach for competence, developing a shared understanding of performance criteria with learners and providing specific, actionable feedback toward the learning goal.^41-43 In the context of CBME, this process has been advocated as a method of using EPAs for teaching.⁴⁴

Supporting Autonomy

Residents will not enter training ready to make sense of EPA performance criteria independently or seek feedback purposively. In assessment for learning, autonomy is supported when teachers gradually provide learners the responsibility of seeking out more feedback to self-regulate their learning as they show evidence of internalizing performance criteria.⁴¹ If we apply this framework to CBME, it becomes clear that entrustment marks the beginning, not the end of competence,⁴⁵ a signal that a resident is sufficiently self-regulating to set their own goals for developing expertise. Program evaluations of CBD advocate culture change to realize the implications of this transformative paradigm for teaching as well as assessment.^17,19

Holistic Change

However, it will not suffice to develop a culture for coaching learners in CBD if burdensome numbers of required EPA assessments are not addressed.^14,15 Residents may feel more supported by their teachers, but autonomy and competence will be thwarted if the focus is directed toward unrealistic or irrelevant expectations, unrelated to individual learning needs.^1,6,27,34,35 Likewise, tailoring EPA requirements to individuals and contexts will be insufficient if logistical and technical challenges to completion of assessment forms continue to be a barrier to receiving actionable feedback.^14,15,19,46 Finally, our results demonstrate that the drive to acquire successful EPA observations to receive a positive report from the competence committee is not “low stakes” for residents. An approach to entrustment by numbers in CBD has placed undue stress on residents, replacing an ethos of support for the development of competence with a surveillance system based on performative metrics. We must address the challenges of assessment burdens in CBD comprehensively because the competence and well-being of learners is at stake.

Limitations

This study provides contextual insights on resident experience of assessment in surgical and anesthesia programs in CBD. The findings are suggestive for other CBD training programs, as well as other contexts of CBME implementation, but must be interpreted with this caveat. Furthermore, as in any interview-based study, participants opted in because they had experiences they wanted to share, which may skew the results toward more negative impressions of CBD. Observational research exploring how CBD translates to practice would offer further elaboration.

Conclusions

This study contributes a contextual understanding of how resident assessment burdens in an approach to CBME emerged as unmet needs for autonomy, relatedness, and competence, with unintended consequences for well-being and intrinsic motivation. SDT offers strategies for meeting these needs to support well-being and the development of competence.

Funding Statement

Funding: This research was supported by a Faculty Support for Research in Education grant awarded by the Schulich School of Medicine and Dentistry, Western University.