Understanding the near-peer relationship: resident perspectives around a novel on-call workplace-based assessment

Abstract

Background

Workplace-based assessment (WBA) is a critical component of competency-based medical education (CBME), though literature on WBA for overnight call is limited. We evaluated a WBA tool completed by supervising subspecialty trainees on paediatric residents during subspecialty overnight call, for usefulness facilitating feedback/coaching in this setting.

Methods

Web-based surveys were sent to residents pre- and post-WBA tool implementation monthly for four months (August-December 2018), exploring feedback frequency, Likert-scaled opinions of tool feasibility/usefulness facilitating feedback, and qualitative experiences. Assessor comments were categorized as actionable/non-actionable. Quantitative data was summarized using descriptive statistics. Qualitative data was coded to identify themes.

Results

Total response rates averaged 41% (total 25 responses, average five respondents/12 residents on-call each month). Post-implementation (n = 16 responses), a non-sustained trend of increased Medical Expert feedback was observed. Residents were generally divided or disagreed on tool usefulness facilitating feedback and feasibility. Comments contained actionable feedback in < 10% of completed WBAs. Qualitative analysis revealed barriers to tool-facilitated coaching including: feedback quality and setting/environment, role of senior near-peer as assessor, interpersonal burden in encounters, and tool-specific issues.

Conclusions

Increasing frequency of WBA tool completion is not sufficient to achieve CBME goals. Factors impacting feedback/coaching within the resident/near-peer dyad must be addressed.

Introduction

Implementation of competency-based medical education (CBME) has been a major transformation within Canadian medical education.¹ CBME uses workplace-based assessments (WBA), in which teachers observe clinical encounters to provide coaching and to assess achievement of competencies, which informs entrustment decisions.² Quality of coaching, including rapport building, is a key factor in WBA success.^3–5

Despite residents spending substantial time in the overnight on-call setting, there exists a gap in our understanding of WBA in this unique educational setting.^6-8 Existing literature focuses on surgical and radiology residents,^9,10 with emerging work in internal medicine,¹¹ but none to our knowledge in paediatrics. As residents are often supervised by near-peer assessors (defined as peers at least one year senior) during overnight call,¹² there is a further gap in our understanding of this assessment relationship within CBME. While the value of near-peer coaching has been described among faculty,¹³ it has not been well studied amongst residents. ¹³

This study aimed to understand the perspectives and benefits of using a WBA tool in facilitating coaching for overnight call. Through our analysis, we explored factors that affected feedback quantity and quality, and the assessment and coaching model.

Methods

Setting

In this core paediatric residency within a large Academic Health Sciences Centre, second-year paediatric residents cover overnight call for multiple subspecialty services in a night float model (~four shifts/month), supervised by subspecialty trainees. Some subspecialty trainees remain in-house while others provide home-call coverage, communicating with residents by phone. Generally, residents are encouraged to contact the subspecialty trainee, rather than faculty, who support subspecialty trainees in a hierarchical manner. This process is evaluated through a Continuous Quality Improvement (CQI) process with feedback elicited throughout the year and via year-end resident retreat reports. Residents consistently requested increased feedback around their on-call work.

WBA Tool

Aligned with CBME implementation and to address the above mentioned feedback gap, we designed a web-based WBA tool adapted from an existing Royal College tool used in Internal Medicine, which included specific milestones, an overall entrustment rating, and comment boxes for feedback within CanMEDS¹⁴ Medical Expert and Leader domains (Appendix A). These roles were selected as competencies in these domains were mapped to the on-call training experience.

The tool was accessible on any electronic device via our institution’s online assessment platform for WBAs (Elentra®). WBA tools could be completed with resident and assessor together on a device, or sent by residents to assessors via email. The WBA tool was to be completed once per shift. WBAs could be initiated by resident or supervisor (subspecialty trainee or staff).

Study design & ethics

Anonymized web-based surveys were sent to residents via email once before tool implementation and monthly after implementation for four months (August-December 2018), to compare before and after tool implementation. Surveys measured resident-reported number of call shifts that included actionable feedback versus total number of shifts worked. Actionable feedback was defined as feedback highlighting specific guidance/advice. Likert-scaled opinions (5-point) were obtained on tool usefulness in facilitating feedback in Medical Expert/Leader roles and feasibility within post-call workflows. Demographic data were not collected, to protect anonymity.

Narrative comments were collected using open-ended prompts exploring how feedback was received on-call and inviting suggestions to improve WBA tool effectiveness.

Comments within completed WBAs were analyzed by one research team member (JJ) and classified as actionable/non-actionable based on inclusion of feedback that a resident could reflect and act upon. Data were presented in aggregate “yes/no” format due to their confidential nature.

This study was approved within the Hospital for Sick Children as a Quality Improvement (QI) initiative. Potential participants were informed about the study prior to its commencement; participation was not required, and consent was implied with completion of survey questionnaires.

Quantitative and qualitative analysis

Quantitative data were summarized with descriptive statistics. Qualitative data were analyzed through a qualitative thematic analysis.¹⁵ Responses were first coded independently by two authors (AL, JJ) using an inductive approach to identify themes. Authors then met to discuss codes and themes to develop shared understanding. Consensus was achieved around themes using codes identified and thematic saturation was reached once no new themes were identified across the dataset.¹⁶

Results

Quantitative results

A total 25 responses were received over the study period, with an average 41% response rate (average five respondents/12 residents on-call each month). Resident-reported frequency of call shifts including actionable feedback over the preceding month suggested a non-sustained trend towards increased feedback following WBA tool introduction (median one shift with actionable feedback/four shifts worked). There were no differences between Medical Expert and Leader domains.

Residents (n = 16 post-tool implementation) were divided between agreement (38%) and disagreement (31%) on tool usefulness facilitating Medical Expert feedback but disagreed that it was useful facilitating feedback on the Leader domain (50%). Residents disagreed that the WBA tool was feasible to use within post-call clinical workflows (63%). Analysis of WBA comments demonstrated that < 10% contained actionable feedback within Medical Expert/Leader domains (Figure 1).

Figure 1.

Distribution of feedback within completed WBAs

Total n = 64 assessments completed of possible 196 (33%). Examples of non-actionable feedback: “great work,” “nice working with you,” “keep it up” Examples of actionable feedback: ““continue to loop back with nurses when you put orders in after hours to close the loop”, “make sure to always reach out to seniors for critical values”

Qualitative results

Our qualitative data provided a more nuanced picture of factors influencing assessment success and revealed themes highlighting challenges in completing WBAs during call. An overview of themes with representative quotations can be found in Table 1 and are outlined below.

Table 1.

Overview of themes within qualitative analysis around barriers to WBA tool usefulness and feasibility

Themes	Sub-themes	Selected Quotations
Feedback Environment and Daytime Handover Structure	Feedback occurring overnight/in real-time	“[I received feedback] from the overnight fellow when I ran a specific issue by her.” “[I received feedback] over the phone during midnight conversations.”
	Daytime handover structure as barrier to feedback post-call	“…Handover is done in the AM via text or phone call and so I had to find a way to meet the fellow after I had handed over to all of the other teams.”
Subspecialty Trainees as Assessors	Lack of availability to provide feedback	“It is difficult to always find the post-call fellow to do the evaluation.” “I had to find a way to meet the fellow after I had handed over all of the other teams.”
	Lack of understanding/awareness of WBA tool	“Most time is taken explaining to fellows what this tool is.”
	Paediatric resident autonomy from subspecialty residents/fellows	“We make a lot of independent decisions overnight.” “There’s times where I don’t interact all that much with the fellow.”
Interpersonal Burden	Assessment/feedback encounter as burden	“I’d have to ask them to stay later.” “Fellows were a bit upset at the length of the tool.”
	Burden amplified by subspecialty resident/fellow assessor being post-call	“The questionnaire was way too long for post call sleepy fellows.”
Feedback Quality	Non-actionable feedback	“I didn’t get more than a ‘I agree’ with my plan.”
	Infrequent feedback	“The feedback isn’t too much.”
	Paediatric resident interest in feedback	“Would really appreciate more constructive feedback.”
Tool	Importance of ease of use	“Needs to be…easier.”
	Preference for shorter tool	“Needs to…[have] fewer things to fill in.”
	Issues with tool platform as barrier to use	“Fellows need to have their usernames integrated into the…system…More options to have the evaluation request sent out to be completed at a later date and time”
	Preference for less frequent tool completion	“Reduce the frequency – it was difficult to have it filled out after each shift.”

Feedback environment and daytime handover structure. Residents identified the post-call setting as a challenging feedback environment, with limitations of the post-call handover structure emerging as a recurring barrier. Clinical responsibilities (handing over to multiple services) were prioritized and interfered with obtaining feedback.

Subspecialty Trainees. Residents described how supervising subspecialty trainees, also on-call themselves, were often unavailable post-call to complete WBAs. Resident autonomy overnight led to perceptions of limited value of WBAs by subspecialty trainees who may not have been involved at the time of the decision. Residents also described challenges with subspecialty trainees’ lack of awareness or understanding of the WBA tool’s purpose and how to navigate the tool platform.

Interpersonal burden. Residents suggested a tension around the coaching encounter, whereby the interpersonal relationship between the near-peer assessor and resident affected the quality of coaching. Assessment encounters were perceived as a burden for assessors, with residents avoiding requesting a WBA to “protect” the subspecialty trainee from extra work. Being post-call amplified this feeling of placing a burden on others.

Feedback quality. Residents identified interest in receiving more constructive feedback, but feedback provided to them on-call, both pre- and post-WBA implementation, was consistently described as lacking in quantity and quality.

Tools. Residents identified priorities to improve the WBA tool including shorter length, ease of tool use, decreased expectations for WBA frequency, and improved platform accessibility for assessors with flexibility in time and space for tool completion post-call.

Discussion

Our study is, to our knowledge, the first to capture the perspectives of paediatric residents on a WBA tool to facilitate coaching during overnight call and highlights the tensions that arise from the near-peer assessment relationship in this setting. We found that assessment encounter frequency may increase following the introduction of mandatory WBAs; however, this does not beget high-quality feedback.

Our study identified context- and tool-specific factors that could be adjusted through a CQI process to improve WBA implementation, namely tool length, expected frequency of completion, ease of tool use, and protected time for completion. We also identified the uniqueness of post-call timing as a challenge for prioritizing WBA completion.

Of interest, we gleaned insight regarding the unique role of a supervising senior trainee as near-peer assessor during overnight call. The “interpersonal burden” imposed on the resident/near-peer dyad during WBAs echoes interpersonal influences described in studies of the senior (internal) medical resident (SMR) as near-peer assessor, whereby an SMR’s focus on team dynamics is in tension with authentic assessment.¹¹ We note this interpersonal burden is exacerbated post-call, as residents are acutely aware of the “burden” of the assessment ask when there are conflicting clinical (e.g. handover) and wellness (e.g. getting home) priorities. As CBME reaches multiple levels of training, consideration must be given to how to balance a senior trainee’s dual responsibilities to assess and be assessed within the same on-call experience.

Assessment strategy implementation must account for the substantial amount of coaching provided by near-peers within academic institutions. While faculty development has been a focus within CBME,^17,18 specific senior trainee education targeting coaching skills, managing the near-peer relationship in assessment, and understanding WBA tools is critical. Coaching skills developed among trainees will follow to faculty positions to sustain CBME efforts long-term.

Similar to others,¹⁹ we found that the quantity of WBAs does not equal quality. Effectively delivered, actionable feedback is a key tenet of the model.^20,21 All educators, including near-peers, need to understand the tension between coaching and assessment. Where a resident’s priority is on being perceived as competent rather than seeking out coaching for growth,^22,23 summative assessments may, in fact, be a barrier to learning. Sawatsky et al. distinguish between assessment of learning and assessment for learning²⁴. Our work suggests that these are in tension with one another, whereby increases in assessment for on-call trainees driven by CBME may increase opportunities to assess abilities, but not necessarily to increase the coaching desired. If we are to benefit from CBME assessments as vehicles to promote resident learning–assessment for learning–ongoing effort on developing coaching skills with consideration of near-peer relationships in academic institutions is important.¹⁹

Limitations must be considered when interpreting our findings. Our study was conducted at a single centre and may not be generalizable to other programs; however, lessons may be relevant where institutions have similar call structures using near-peer assessors. While reporting bias may have influenced survey-measured feedback rates, the same tool was used to compare pre- and post-tool rates, so we presume any bias would have affected both rates. Though linking assessment and survey data would have provided an interesting perspective on results, this was not done to preserve respondent confidentiality.

Conclusion

Introducing a WBA tool is not a sufficient catalyst to promote the CBME goals of coaching. To improve coaching in an assessment encounter, concerted, intentional action is required including developing feedback quality, coaching relationships and skills, and delineating setting-specific roles of senior near-peers in assessment. Ongoing work and study will be critical to ensure that we are able to derive the value and learning we hope to achieve from both WBA encounters and CBME adoption more broadly.

Appendix A. Sample post-call WBA tool

Conflicts of Interest

None

Funding

The authors received no specific funding for this project.