Main themes and sub-themes	Sample Participant Quotation(s)
1. Tracking changes in overall assessment metrics Figures 2, 3a-d	The--yeah, in terms of acquired and expired, that’s nice data to have and see where we’re doing as a program and gets me a sense of nice piece of data to go to a department meeting or talk to faculty about and say, “Look, as a group, we’re not doing as good a job of this,” and then, that can give me some, likely, some support at higher levels.
1.1 Program-wide EPA acquisition Figure 2	super helpful ‘cause we’ve already with our curriculum blueprint figured out where the EPAs should be getting filled out. So this is a great way to look at it and see if there’s EPAs that are not getting filled out that should be and then trying to sort that out.
1.1.1 Annual EPA completion Figure 3a	Is there a way we can overlay year to year? ‘Cause that would be--that’s really what we wanna see, what’s changing like what interventions we may have provided and what’s working and what’s not working.
1.1.2 Monthly EPA completion Figure 3c	Monthly distribution, yeah, that’s good, just to make sure it’s across the board this is just integrated with our practice and not just sorta a prior to performance review exercise and then. John4 I like the monthly distribution as well. Like December’s obviously expected just because I know people take holidays and things like that but if we see things kind of hop off towards the end of the year we can do extra prompting to the residents that like we see you guys are really good in July but then you drop off later in the year. We need to keep that consistent and things like that, and just, kind of see if it’s specific rotations during those times versus if it’s just more kind of resident behavior.
1.1.3 Modifiers of EPA completion	I would say honestly just for the year. Like the average per resident honestly because we have more residents this year so obviously our EPA count’s gonna go up… Yeah. And just at the end of year and say like how many EPAs did our residents get? And it’s like okay they’re averaging this many per month or this many for the year, and this year they’re at.
1.2 Narrative Feedback Quality Figure 2, 3d	What is the expectation for the timeliness of it? Or what should be the content of an EPA and what makes a good EPA? What makes an EPA great, what the residents perceive as being valuable? What does a program or CBD committee perceive to be valuable? So, as you do those quality metrics, maybe get into some of that content, it’d be useful and you can say well, could this be—’cause we’re already being assessed on performance and teaching, right, but the metric’s very crude… So, I think there’s an opportunity to provide that feedback, both to faculty and, at some point, to, obviously, the program and CBD committee, but also to Department Head ‘cause department heads—my last assessment, I’ll share with you and it could be part of this recording. My last assessment, pretty well every assessment, I think the teaching, the quality of teaching, is not really discussed at all. It’s just glanced over.
1.2.1 Word Count Figure 2, 3d	I like the feedback word count. That’s good too because that helps I think people strive for it; it can be done. And if you go down, what all did you have down at the bottom there? Oh yeah this. This, like you say this is a little harder to look at because not every program’s consistently uploading everything. Like I suspect <program> hasn’t uploaded anything for a while, but. But the top I think is very useful.
1.2.2 Quality Evaluation	Obviously we talked before about the quality, the narrative comments beyond word count if that were possible. And then the correlation between entrustment score and comment, if it makes sense, if there was some way to measure that and give people feedback on it, that would be useful.
1.3 Expired EPAs Figure 2, 3a	So I have to admit, you know again going back to <specialty 1>, the expired is very helpful. Because residents will often say “oh they just all expired, we didn’t get any” well it’s like, less than 20% is not bad. Because there are gonna be some. So I think this is a very nice visual.
1.4 Distribution of EPA entrustment ratings Figure 2, 3b	Probably not related to the program, itself, but we have pretty significant either leniency bias or with how we entrust things or a substantial proportion of the work that we do is relatively easy to do and capturing the stuff that’s harder to do and harder to entrust is less frequent. I’m not sure which of those two phenomena is at play, but the fact that the overwhelming majority of our EPAs--what is it? … like 80% of our EPAs or more, are scores of four and five is, it worries me a bit, but good information to have.
1.5 Time to completion of EPAs	Because if I notice one program is always doing them 14 days later that obviously needs some education because that’s just not, overly helpful. And if we have one program that’s always getting them done within 24 hours we need to figure out what they’re doing right.
1.6 Resident versus faculty generation of EPAs	Yeah, the other one I’d like to see, I think I’ve talked to you about before, is resident generated EPAs versus faculty generated EPAs. I’d like to capture that data and I think that’d be useful in a program evaluation level, just to see are the residents carrying CBD for the program or is the faculty carrying CBD and how’s this working.
2. Comparing metrics to the assessment plan Figure 4	It is useful ‘cause it’s like a nice overview of which EPAs are not being assessed. And then we can then do a deeper exploration such as with your EPA distribution and rotation-specific distribution, it can inform how you use this. Because to tick through this and look at every single one is not gonna be really meaningful. It’s unnecessary, right? But to find an EPA, there might be three or four of them, they’re like, “Hmm we’re not getting a lot of those this year.” Let’s look at those four and let’s dive deep into those four and figure out what our challenges and solutions could be. It is useful to see it year over year. It kinda talks about your implementation and then sustainability versus fatigue, to see if whatever orientation efforts you did with your faculty are being sustained or further efforts are needed to make sure that's it’s done. So, yeah, it’s quite useful to see year to year.
2.1 Underrepresented EPAs Figure 4	No, I think that’s actually quite—it’s very useful because there may be lots of reasons why a certain EPA is not being delivered, but there may be some structural reasons. Maybe the rotations underperforming ‘cause resources should be—whatever the case is, right? So, maybe that’s an appropriate sort of—the particular rotation should move to a different education experience. So, I really like the graphs ‘cause you can see that C3s, and fours, and fives, and sixes, some of these, in that program, were obviously not completed. So, it’s really worth asking why that is and find out reasons. Yeah, that’s very useful.
2.2 Overrepresented EPAs Figure 4	I would just wanna make sure, with this data, that if we were over target on every EPA, that we weren’t sort of punishing ourselves to say that, well, they’re not balanced, ‘cause I don’t really care if they’re unbalanced, as long as they get the minimum target for each of them. So, if they needed 10 resus, but they got 15 and they needed 20 undifferentiated patients and they got 30 and they needed contributing to the teamwork which was ten and they got ten, then to me, that’s perfect.
3. Evaluating rotation performance Figures 5a-c	I think what program directors have been asking for is the ability to see where EPAs are getting done. So like from a program evaluation, like an individual discipline [inaudible, 00:13:21] I think this would be very helpful. ‘Cause right now they can’t really see that. And so you’re right, we’re sort of trying to go through a multi-curriculum map and make sure it’s mapped and then this is sorta your check to see does it actually do what you want it to on the ground.
3.1 Underperforming Rotations Figures 5a-c	And that we can better plan for as we need to adjust the schedule going through. And then EPA count per rotation so that’s more helpful for me from the CBD lead perspective I would say, just because if we hit a block like where we’ve had three residents on <rotation> in the last three months and I filter it out and we have no EPAs completed, that’s helpful for me if I have to send a prompt to any of the faculty on <rotation> or the rotation lead on <rotation> and just be like, ‘hey this is what we’ve noticed over the last couple of months, we’ve had a few trainees rotate, we’ve had some expiry with your faculty’s EPA as well. Could we’- we’ll just send out this email prompting them just to fill out EPAs.
3.2 Overperforming Rotations Figures 5a-c	Cause we have targets for how many we wanna get per rotation and to look at… see oh, <program>? On average, they’re getting eight, right? Which is exactly what we’re hoping for or we’re hoping for four to eight. Same with <rotation 1> and same with <rotation 2> and they’re sort of getting exactly those numbers. So, that’s a nice marker of are we hitting where we want. So, off-service are sort of that four to eight range, great and for our more core rotations, we obviously wanna see--our <program> rotations, we wanna be seeing that 14 to 20 range which is exactly what we’re seeing and then, things like <rotation 3> and <rotation 4>, they’re punching above their weight class, so this would be a nice thing, as part of our six month review, we identify teachers who are sort of over-performing and not necessarily getting a formal award, but just a quick email with a CC to their department head saying, “Just wanted to let you know the residents thought you were a great teacher in the last six months and we really appreciate what you’re doing for our program,” this would be a nice little feather in the cap to email the <rotation 4> program director, department head, and just say, “We did our six month review of our rotations, when our residents rotate through your service, they’re getting an average of 12 EPAs observed per block which is higher than our target of eight. Just wanted to say thank you so much for your commitment to educating our residents,” and that sort of stuff goes a long way for keeping up relationships. Then, if we saw one that was sort of under-performing, not the opposite of an email saying, “You suck,” but trying to explore why that might be happening.
4. Engagement with the assessment metrics	No, I think it’s fine to share that data. I think it would potentially be helpful for like rotations where, or specialties where it’s like you rotate through this but you’re not entirely sure what EPAs you’re gonna have done. You might just use like a shift in counter card because your residents are off-service and you don’t necessarily know what to do. But there’s actually EPAs that the program fills out for their own residents that their faculty are filling out and they’re getting a reliable amount of exposures that are similar, then maybe hey you can actually target this EPA when you’re on a rotation. So thinking of it from that side of things. In terms of it actually leaving to meaningful change I think it’s really a matter of getting to the people that are on the ground. So if it can be shared with other programs and those programs, like the leads or the representatives from them are able to engage with their faculty and actually use the data to change behavior, I think that’s helpful. But I think it’s tricky from that sense. But I wouldn’t be opposed to sharing it in any way.
4.1 Availability	This is what I love about it is that this helps the CCC or the RPC make real-time decisions on where learning experiencing should be provided. And they don’t have to wait for one year of data to come up and get feedback from residents who may or may not speak. So this is truly awesome!
4.2 Notifications of abnormal data	Only if there’s an egregious issue, which of I don’t know if [inaudible] or not, but to make other things for that, that information and things like that, they will not show up here. But, the other things that require immediate action should be sent right away, otherwise three months.