From Practice to Proficiency: Evaluation of a Novel Workplace-Based Assessment (WBA) in a Postgraduate Chemical Pathology Residency Program in Pakistan

Abstract

Introduction

A workplace-based assessment (WBA) model was implemented in the postgraduate (PG) residency program of Chemical Pathology at the Department of Pathology & Laboratory Medicine, Aga Khan University (AKU). PGs were assessed using direct observation of practical skills (DOPS), evaluation of clinical events (ECE) and case-based discussion (CBD) on a virtual learning environment (VLE) platform.

Objectives

To evaluate WBA frequency, case mix, feedback, and satisfaction levels of faculty and PGs of Chemical Pathology at AKU.

Methods

Data from January 2019 to June 2023 was assessed. Tool utilization and case mix frequencies were calculated. PG and faculty satisfaction levels, as well as feedback and discussion time, were averaged. A thematic analysis was conducted on descriptive comments.

Results

Out of 911 WBAs attempted, 79.1% (n=730) were CBDs, 10.8% (n=98) were DOPS, and 9.1% (n=83) were ECEs, showing a well-distributed case mix. Average satisfaction levels for CBD, ECE, and DOPS among both PGs and faculty were 8.38, 8.48, and 8.59, and 8.20, 8.36, and 8.46, respectively. Faculty feedback averaged 8.40, 8.65, and 7.85 minutes for CBD, ECE, and DOPS, respectively. Discussion times averaged 9.37, 9.52, and 13.36 minutes for CBD, ECE, and DOPS, respectively. Suggestions for development were noted in 20.82% (n=225) of CBDs, 21.69% (n=18) of ECEs, and 16.32% (n=16) of DOPS. Positives were documented in 40% (n=292) of CBDs, 28.92% (n=24) of ECEs, and 7.14% (n=7) of DOPS.

Conclusion

This study evaluated a web-based WBA model in chemical pathology training, suggesting its applicability in diverse pathology specialties and regional training programs.

Introduction

Workplace-based assessment (WBA), initially labelled as a formative or ‘assessment-for-learning’ was introduced in postgraduate medical education (PGME) as part of a global shift towards competency-based medical education. Its primary aim is to steer postgraduate (PG) trainees towards specific learning objectives [1]. Formative assessment does not simply assign grades to trainee performance at specific points in the curriculum. Instead, it is curated to seamlessly integrate into the instructional process with the overarching goal of fostering and enhancing learning [1]. One significant advantage of WBAs over other evaluation methods, such as multiple-choice questions or objective structured clinical examinations (OSCEs), is their ability to facilitate real-time assessments. With this approach, the supervisor directly observes the trainee in a specific aspect of clinical practice, thereby increasing the validity of WBAs compared to other assessment methods [2]. However, evidence suggests that the greatest impact of WBA lies in providing observation-based feedback making it a powerful tool for positively impacting learning behaviours by encouraging proactive pursuit of feedback [3-5]. Despite these appeals, concerns regarding the practical feasibility of WBA persist as the implementation of WBA is seen as an additional demand on the trainer and trainee in busy clinical settings [1, 4]. To complete post graduate medical training in chemical pathology, guidelines of the College of Physicians and Surgeons of Pakistan (CPSP) mandate the completion of a 4-year training program. Responsibilities of PG trainees in Chemical Pathology mainly include examining preanalytical, analytical and post analytical aspects before validating any biochemical result, investigating interferences on test results, conducting clinical audits for quality and process improvements, and collaborating with consultants in performing procedures, provocative tests and assessing new tests for potential introduction in service to name a few [6-8]. Concerns about resident involvement and postgraduate medical education (PGME) in Chemical Pathology have been brought up by developments in laboratory medicine. Multiple surveys of pathology residents have shown that Chemical Pathology’s supporting learning environments are inadequate. Additionally, after finishing their residency program, many graduates have stated feeling inadequate to practice clinical chemistry on their own [9, 10]. These issues are further augmented by the concern that PG trainees in Chemical Pathology are seldom observed and given feedback, whereas feedback is central to learning and lies “at the heart of medical education [3].” To address this, a web-based WBA model was implemented in January 2019 in a postgraduate residency program of chemical pathology at Aga Khan University (AKU) using a virtual learning environment (VLE) platform, Moodle, to assess the knowledge, skills, professionalism, and critical thinking of PG trainees while providing comprehensive feedback. Validated WBA tools were chosen using Norcini AMEE guide and modified according to the needs of chemical pathology training [8]. The objective was to facilitate ongoing evaluation and learning for trainees in high-volume laboratory environments by seamlessly integrating assessment into their daily work activities. This approach ensures that trainees receive timely feedback on their performance, allowing them to identify areas for improvement and further development. By embedding assessment within the context of their work, WBA enables trainees to apply theoretical knowledge to practical situations, fostering a deeper understanding of key concepts and enhancing their skills. Moreover, the continuous nature of WBA promotes a culture of self-reflection and improvement, empowering trainees to take ownership of their learning journey. Overall, WBA serves as a powerful intervention in medical education, offering a holistic approach to assessment that aligns closely with real-world clinical practice. A grading scale ranging from 1 to 6 was employed, where ratings of 1 to 2 indicated performance below expectations, 3 denoted borderline performance, 4 signified meeting expectations, and ratings of 5 to 6 indicated performance above expectations. In instances where ratings fell within the range of 1 to 3, focused faculty feedback was provided, necessitating re-training within the respective domain, followed by reassessment. Furthermore, a predetermined minimum number of WBAs was established for each residency year. These assessments were required to be completed and documented within the VLE platform, constituting an integral component of the promotion criteria. It is widely acknowledged that the success of innovations in medical training programmes hinges on promptly identifying opportunities and pitfalls [11]. Thus, the goal of our present study is to evaluate the data recorded on VLE since the introduction of WBA five years ago and assess its efficacy and long-term feasibility. We intend to analyze the frequency of WBA tools usage, the regularity of feedback provided to the PGs, the distribution of specialties among participating PGs, the variety of cases and procedures discussed, the average time taken for feedback and discussion, satisfaction levels of both PGs and assessors, and examine qualitative feedback to identify potential gaps and areas for improvement.

Methods

A retrospective study was conducted at the Section of Chemical Pathology, Department of Pathology and Laboratory Medicine, AKU, from September 2023 to January 2024. The WBA tools most relevant to Chemical Pathology training including direct observation of practical skills (DOPS), evaluation of clinical events (ECE) and case-based discussion (CBD) were implemented. All PGs, including fellows, registered in the chemical pathology training program at AKU and those rotating in the section of Chemical Pathology from other Pathology, Pediatric or Medicine specialties were eligible for WBA. Assessors included faculty, senior technologists, and sectional managers of the Chemical Pathology section at AKU [8]. Data was extracted from all the responses following the completion of each WBA form on Moodle between January 2019 to September 2023. The permission of AKU’s ethical review committee was sought before the initiation of this project. The cumulative number of WBAs conducted since January 2019 was calculated, including the utilization of each WBA tool on an annual basis. Averages were computed for the feedback and discussion duration associated with each tool, categorized by year. To assess the range of topics discussed in WBAs, an analysis was performed on all cases covered in each assessment, with frequencies calculated accordingly. Furthermore, frequencies were determined for the procedures and instruments evaluated using the DOPS tool separately. Satisfaction ratings from students and faculty regarding the WBA process, recorded on a 10-point Likert scale in each WBA form, were averaged. To gain insight into the participation of PG trainees in WBAs to date, the total number of participating trainees, along with their respective specialties, was also analyzed. Each WBA form featured two sections for qualitative feedback to PG students: one for developmental suggestions and another prompting comments on particularly positive aspects of the conducted WBA. The frequency of feedback provision for each WBA tool was assessed. Additionally, a qualitative analysis was conducted on the feedback comments provided in both sections. Conceptually similar statements were grouped together and systematically organized under thematic categories.

Results

Our collected sample represents a total of 911 WBA forms out of which 80.1% (n=730) were CBDs, 10.8% (n=98) were DOPS, and 9.1% (n=83) were ECEs.

Figure 1 illustrates the case mix and percentage distribution of topics covered under all WBA tools. Table 1 represents the procedures and instruments upon which DOPS was demonstrated.

Figure 1:

Percentage distribution of WBA case mix topics covered from 2019 to 2023 at the section of Chemical Pathology (n=911)

Table 1:

Procedures and instruments upon which DOPS was demonstrated from 2019 to 2023 at the section of Chemical Pathology (n=98)

	Instrument/Procedure (DOPS)	Frequency
1.	Procedures	19
	Growth hormone insulin test	4
	Breath hydrogen test	6
	Sweat chloride test	9
2.	Lab management	11
	Complaint handling	1
	Recovery study	1
	SOPs preparation	1
	File maintenance	7
	Preparation of inventory binder	1
3.	Instrumentation	57
	Integrated clinical chemistry and immunoassay analyzer	1
	Chemiluminescence immunoassay	4
	Isoelectric focusing electrophoresis	1
	Gas chromatography mass spectrometry	1
	ABG analyzer	4
	Centrifugation	6
	Stone analyzer (FTIR)	24
	Electrophoresis	2
	Analytical balance	3
	Total lab automation	1
	Semi-automated clinical chemistry analyzer	1
	Osmometer	4
	Immunofixation electrophoresis	4
	Point of care testing	1
4.	Method validation and verification	7
	Linearity study	1
	Precision study	1
	Prolactin (PEG method)	5
5.	Lab Safety	4
	Biological and chemical spill	4

DOPS proved to be the most efficient at feedback provision with a duration of 7.85 minutes compared to CBD (8.40 minutes) and ECE (8.65 minutes). However, DOPS discussions were lengthier, averaging at 13.36 minutes compared to CBD (9.37 minutes) and ECE (9.52 minutes). Trends for these variables across the years are shown in Figures 2(a), 2(b) and 2(c) for CBD, ECE, and DOPS, respectively. The average PG satisfaction levels with CBD, ECE, and DOPS were found to be 8.38, 8.48, and 8.59 on a 10-point Likert scale, respectively. On the same scale, faculty satisfaction levels for CBD, ECE and DOPS averaged 8.20, 8.36, and 8.46, respectively. Since 2019, 47 PG trainees have taken part in WBAs. Out of these, majority were rotating trainees from histopathology (28%), followed by hematology (23%), microbiology (17%), adult endocrinology (13%), and pediatric endocrinology (4%). The trainees of chemical pathology represented 15% of all participants. Regarding feedback comments, suggestions for development were provided in 20.82% (n=225) of CBDs, 21.69% (n=18) of ECEs, and 16.32% (n=16) of DOPS. Positive aspects of the conducted WBA were highlighted in 40% (n=292) of CBDs, 28.92% (n=24) of ECEs, and 7.14% (n=7) of DOPS. The descriptive analysis of feedback comments on the three WBA tools yielded the following themes presented in Table 2.

Figure 2(a).

Average time taken for discussion and feedback in CBD

Figure 2(b).

Average time taken for discussion and feedback in ECE

Figure 2(c).

Average time taken for discussion and feedback in DOPS

Table 2:

WBA tools yielded the following themes.

	ECE	DOPS	CBD
Suggestions for Development	Research skills enhancement Diversification of knowledge base Quality control and methodology understanding Improvement in technological competence	Skill practice enhancement Revision of SOPs Technical and instrumental proficiency development	Improvement in the understanding of laboratory techniques and methodologies Further improvement in clinical and pathophysiological knowledge Professional development and research collaboration
Positive highlights	Communication and proactivity Clinical competence and case management Comprehensive theoretical understanding	Professional conduct Test knowledge proficiency	Effective selection, presentation, and preparation of cases Proactive learning opportunity and knowledge demonstration

Discussion

To our knowledge, our study represents a pioneering effort in Pakistan by introducing a WBA model within a chemical pathology training program. Although formative assessments are popular in residency programs across various medical specialties, their utilization in pathology subspecialties remains relatively scarce [12]. Given the modest size of the chemical pathology residency program, with a limited number of trainees nationally and globally, our study offers valuable insights. The growing volume of WBAs conducted over successive years adds to the validity of our study findings. Moreover, during the initial phase of our model implementation in 2019, faculty and PGs qualitative feedback on perception of WBA was taken via interviews. aimed at identifying challenges and areas for improvement [8]. This feedback was instrumental in refining the processes, ensuring enhancements were made prior to its official implementation in the residency program. Furthermore, as faculty time constrained was a leading challenge and to facilitate, few senior lab scientists and manager were also included as evaluators in the program particularly for bench skills. Our evaluation revealed that the WBA model is being successfully employed in the chemical pathology residency program at AKU. This is evident by its frequent use, feedback and discussion sessions that are convenient for both students and faculty in the clinical workplace, and a diverse case-mix. Furthermore, the high satisfaction levels expressed by both our assessors and PG trainees, alongside the substantial positive highlights documented in faculty feedback, affirm the successful integration of WBA into our chemical pathology training program and day-to-day laboratory practice. We discovered that CBD was the most frequently used WBA tool. This compares to the findings from a UK study where NHS trainee doctors and trainers preferred CBD over DOPS as they facilitate “real-learning” as opposed to DOPS which are merely “tick-box exercises [13].” In accordance with prior studies, our research also supported the value of WBAs in enabling structured narrative feedback and supporting learners with suggestions for further development [14-18]. In contrast, a relatively understudied role of WBAs is in their potential to assess entrustable professional activities (EPAs) in graduate training programs. Entrustable professional activities (EPAs) are quickly taking over as a key assessment tool in the implementation of competency-based medical education. Proposed by ten Cate in 2005, EPAs are “units of professional practice, defined as tasks or responsibilities to be entrusted to the unsupervised execution by a trainee once he or she has attained sufficient specific competence [19]. Thus, assessments based on EPA generate summative entrustment decisions, deducing an individual’s ability to perform under a specified level of supervision [20]. Given the success of our current WBA model, further steps can be taken to upgrade it into an EPA-based WBA tool designed to monitor entrustment decisions, supervision levels, and the autonomy of PG trainees. A similar model implemented for fellows of pediatric critical care medicine provided compelling evidence in support of an EPA-based WBA tool to improve learning by benchmarking and monitoring entrustment levels [21]. Furthermore, we successfully demonstrated the feasibility and advantages of incorporation of a virtual learning environment platform, Moodle, as a core part of PG training. For any healthcare institute, a VLE can offer a platform where students’ performance can be recorded and monitored across the years. Especially in the local context, limited literature exists for exploring the scope and potential of e-learning in medical education [22, 23]. This is particularly unfortunate because evidence suggests that Moodle has been successfully used in teaching acute medicine, urology, physiology, dental radiology, thoracic surgery, and ethics [24]. Therefore, there is an imminent need for medical institutes to consider online learning platforms to support and encourage innovations in medical education. The outcomes of the descriptive analysis revealed many aspects of the narrative feedback provided to the PG trainees. The positive aspects of our trainees were highlighted by assessors which mainly included clinical competence, proactive approach to learning and a strong theoretical foundation. Additionally, suggestions for development were provided, serving as valuable pointers for focused learning for trainees. Our analysis across all three WBA tools demonstrated common developmental areas including the need to develop technical proficiency, cultivate a broader knowledge base and explore opportunities for professional development such as research collaborations. These findings will help the faculty in providing focused assistance and guidance to trainees for a more personalized learning experience to build their proficiency in these identified weaker areas. This also aligns with cognitive theorist’s belief that emphasizing the disparity between an individual’s current performance and the desired level of performance serves as an important catalyst for learning [25]. To further enhance the feedback’s efficacy, assessors should consider fostering recipient reflection-inaction. This would ensure that the trainee’s reflections inform the formulation of an action plan which would emphasize self-monitoring and self-regulated learning, while fostering a supportive teaching and learning environment [26, 27]. To evaluate the genuine impact of our WBA model on a graduating trainee, our way forward is to conduct a study involving graduates from our training program who were exposed to this model to assess the effectiveness of the WBA approach in enhancing their real-world practice. However, as with any novel approach, our model had its limitations and gaps. We found that the frequency of documented feedback was considerably less than the number of WBA forms attempted. This could potentially mean that there was real-time verbal feedback which was not mentioned in the WBA forms. Similar issues have emerged internationally, with many institutions now opting for WBA smartphone and tablet apps which ensure real-time completion and uploading of feedback [1]. In addition to documentation issues in feedback, we identified several other areas with incorrect documentation where assessors did not consistently complete the forms in their entirety or occasionally filled them out incorrectly. Specifically, some assessors failed to explicitly state the name and type of topics being discussed or document the agreed action with the student as required. This discovery offers the opportunity to discuss and tackle these concerns among the assessors.

Conclusion

Overall, our study contributes valuable insights to the ongoing discourse on effective assessment methodologies in competency based medical education. This research was the first of its kind to successfully evaluate the implementation of a web-based WBA model in a chemical pathology training program at a tertiary care facility. Our findings imply that there is potential for replicating this tool in other sub-specialties of pathology, as well as other post-graduate training programs both locally and regionally. Additionally, they underscore the advantages of technological integration and utilizing e-learning platforms in medical education, highlighting the numerous opportunities they can offer to medical institutions.