Table 1.

Summary of Major Themes

Themes	Quotes
1. Value of feedback for residents	“They’re generally in alignment with informal feedback I’ve had. So I haven’t really changed any behavior ‘cause it’s been in the moment when I’ve received the feedback and I’ll adjust my behavior at that time for a specific skill, for example. So what the EPA says after everything is completed doesn’t change what I’ve done during that assessment” (Program 1; Interview 1) “I think for me the CBD is better than the ITER [In Training Evaluation Report]. The ITER sounds a bit more generic and often at the end of the clinic, the staff person may not necessarily remember every single detail thing to improve on, whereas CBD right away it’s very directed to actionable correction measures that you can do to each specific. It’s very specific detail-oriented, whereas ITER is like a big picture of things. To me that’s what it seems like” (Program 4; Interview 9)
2. Resident strategies for successful EPA observation completion	“In a weird way, sometimes as a learner you’ll probably try to target the cases that you felt like you did very well on so that you’ll get the success on a EPA. And there’s probably lots to actually talk about in terms of where you should be going forward and the actual true good learning opportunities and the good feedback sessions would kind of go towards a lot of cases that I have more trouble with but then knowing that I need to get successes on my EPAs, I’d be less inclined to actually get those EPAs done. So no, I think that often I’ve tried to get preceptors to do EPAs that I felt like I would get a success on and then ask for qualitative feedback away from any of the EPA system and that’s where probably time constraints kind of come in a little bit for some people” (Program 5; Interview 11) “You can always swing things a certain way, you can always get EPA from people that you saw you on a good day, you can always use particular EPAs for a different thing that you planned because you had a bad day and you wanna use this really easy EPA or something. There’s ways to get around it” (Program 3; Interview 8)
3. Residents experience challenges	“Certainly staff engagement and understanding of the program is an ongoing challenge. A lot of them are aware of the program now, which is a step forward but many if not most of them still do not feel comfortable actually completing the EPA themselves, and they have poor understanding of the consequences of this, of how to respond to various questions. They don't really understand the implications of each question” (Program 6; Interview 13) “Wording, staff adherence, procedural EPAs if staff are uncomfortable supervising them or if they haven’t done them for a while, I think that’s a big factor, time. It’s true that this should only take 5 minutes, but you add a whole bunch of things that take 5 minutes and then before you know it it’s an hour, it’s like an admission bundle. You’re like oh, EPA should only take 5 minutes, but the admission orders take another 5, 10 minutes and then the note, goals of care, that’s another 5, 10 minutes and then you have 10 admissions, that’s like 3½ hours right there. So it’s not as if things are getting less busy and even for staff if you have four or five residents on and you’re doing an EPA every day or something like that and it’s not just the actual doing the EPA, it’s reviewing, it’s giving feedback, it’s having an active discussion with the resident, all this stuff takes time” (Program 2; Interview 3) “…there’s been a lot of challenges regarding the language used, specifically relating to not so much the milestones I guess, but the actual level of skill obtainment, meaning a lot of physicians will still say, need to be there or I was, you know, needed to be on standby, when in reality they actually didn’t do anything and obviously whatever took place could’ve been done irrespective of them being there. Most people say, I needed to still be there. And so I think sometimes the language has been vague and misleading and so then as a result the assessments have sometimes been inaccurate” (Program 2; Interview 5)
4. Resident concerns regarding CBME	“...this is the irony in all this, that the structure of CBD is great. It’s like the chair is comfy but the seatbelt hurts. The idea of all of it is great but when you all of a sudden strap people in and say well now you have to have this filled out and you actually have to have all these numbers, that’s when you start to go but wait, what happens when all of a sudden we need to slow down a bit or we need to move around a bit?” (Program 3; Interview 8) “It turns it into a bit of a grocery list to be honest with you. Otherwise normally...you read around cases that you have, you read around the physiology and medicine specific to the type of practice you’re in at that time and then also where you study in half day. So you kinda look at bigger picture stuff and then with the kind of advent of EPAs we’re basically just gonna blast ‘em through a list to try and get stuff filled out. And, like I said, it just turns it more into a grocery list than a learning adventure” (Program 3; Interview 6)
5. Resident recommendations to improve existing challenges they face	“So I think assessments should be worded in a way that is less subjective and recognizing that all assessments are to some extent subjective, but language along the lines of the resident completed all key aspects of this particular entrustable activity without any prompting or direct supervision I think is more applicable language than I didn’t have to be there in theory, which probably gets at the same underlying kind of competency and independence but without using that same language that I think some preceptors are reluctant to sign off on” (Program 2; Interview 2) “I think it’s good to stay on top of your EPAs and what’s expected of you because there are so many different ones that you’re expected to have filled out and you do so many different activities on a daily basis, sometimes it’s hard to realize the little procedure, little thing you just did is actually an EPA that could possibly be filled out and just asking staff repeatedly. I know I was hesitant…on asking daily, or weekly, for staff to fill out the EPAs I’d sent out, but then as a result they didn’t get filled out always… [I] feel bad to bug them more” (Program 1; Interview 1)