Workplace-based Assessment; Applications and Educational Impact

Abstract

Workplace based assessment (WPBA) refers to a group of assessment modalities which evaluates trainees’ performance during the clinical settings. Hallmark of WPBA is the element of observation of the trainee’s performance in real workplace environment along with relevant feedback, thus fostering reflective practice. WPBA consists of observation of clinical performance (mini-clinical evaluation exercise, direct observation of procedural skills), discussion of clinical cases (case based discussion), and feedback from peers, coworkers, and patients (multisource feedback). This literature review was conducted on the databases of MEDLINE, EMBASE, CINAHL, and The Cochrane Library. Data were retrieved by connecting Medical Subject Headings (MeSH) keywords: ‘workplace based assessment’ AND ‘mini-clinical evaluation exercise’ AND ‘direct observation of procedural skills’ AND ‘case based discussion’ AND ‘multi-source feedback’. Additional studies were searched in the reference lists of all included articles. As WPBA is gaining popularity, there is a growing need for continuing faculty development and greater evidence regarding the validity and reliability of these instruments, which will allow the academia to embed this strategy in the existing curricula. As of today, there are conflicting reports about the educational impact of WPBA in terms of its validity and reliability. This review draws upon the spectrum of WPBA tools, their designs and applications, and an account of the existing educational impact of this emerging assessment strategy in medical education. Finally, the study reports some educational impact of WPBAs on learning and emphasises the need for more empirical research in endorsing its application worldwide.

Introduction

In the traditional apprenticeship-training models, trainees confront a wide range of health-care problems and, as other physicians practice, they are required to apply their professional capabilities in a competent and skillful manner (1). It is essential that the assessment models should safeguard the safety of patients as well as to offer the opportunity of contextual feedback to the trainee. For plausible solutions, a wealth of assessment tools have been described, many of which are modifications of the conventional clinical long case examination. WPBA entails the evaluation of daily clinical practices employed in the working situation (2). Simply, it is an “assessment of what doctors actually do in practice” (3). WPBA encompasses wide range of assessment strategies that evaluate trainees in clinical settings and provide feedback. WBA has allowed the transition away from the use of numbers-based experience toward a more structured format of assessment (4).

WPBA has been adopted by the UK General Medical Council (GMC) and the Academy of Medical Royal Colleges (AoMRC) for the assessment of performance in the postgraduate medical education (5). Likewise, WPBA is also being used and gaining popularity in the undergraduate medical education (6). GMC elaborates WPBA as assessments for learning (formative), rather than as assessments of learning (summative) (7). Despite this, WBA gathers objective evaluation of a trainee’s competence and performance, providing a generic blueprint of summative functions. This review explores multiple dimensions of WPBA with a view to look into its educational impact on medical trainees and physicians.

Study Design

This search was conducted on the databases of MEDLINE, EMBASE, CINAHL, and The Cochrane Library. Data were retrieved by connecting Medical Subject Headings (MeSH) keywords: ‘workplace based assessment’ AND ‘mini-clinical evaluation exercise’ AND ‘direct observation of procedural skills’ AND ‘case based discussion’ AND ‘multi-source feedback’. Additional studies were searched in the reference lists of all included articles.

The body of information from literature showed a range of categories of WPBA.

Categories of WPBA

A number of WPBA strategies exist, all aiming to assess various facets of trainees’ performance. Their categories and specific indications are summarised in Table 1.

Table 1.

Categories of workplace based assessment and their objectives

No.	Tasks	Tools
1	Observation of clinical performance	Mini-clinical evaluation exercise Direct observation of procedural skills
2	Discussion of clinical cases	Case based discussion
3	Feedback from peers, coworkers, and patients	Multisource feedback (360° assessment) Mini peer assessment tool Team assessment of behaviors Patient satisfaction questionnaire

Observation of clinical performance

Mini-clinical evaluation exercise (mCEX)

In the mCEX, an assessor evaluates a trainee–patient interaction in any healthcare institution. Such clinical encounters are expected to last for about 15 minutes, and the trainee is expected to conduct a focused history and/or physical examination within this stipulated time (8). At the end, trainee suggests a diagnosis and management plan, the performance is graded by using a structured evaluation form, and then constructive feedback is provided. Assessors use a nine-point Likert scale ranging from ‘unsatisfactory’ to ‘superior’(9). This provides six domain-specific ratings and one final rating of clinical competence. Trainee undertakes around six clinical assessments during the year, with a different assessor for each session.

Nair et al surveyed a group of international medical graduates for the acceptability, reliability, and feasibility of mCEX and reported that about 50% graduates were either satisfied or very satisfied with this assessment strategy for learning (10). Other studies examined the educational impact of mCEX in anesthesia training and showed that majority of trainees (and their evaluators) were satisfied by the schedule of assessments and the quality of feedback offered (11).

Direct observation of procedural skills (DOPS)

DOPS was introduced by the Royal College of Physicians and now forms an integral component of WPBA for doctors in the foundation year and those in specialist training (12). It was specifically designed to assess procedural skills involving real patients in a single encounter. During DOPS, an assessor evaluates a trainee conducting a procedure as a part of his/her routine practical training against a set criteria, which is followed by a face-to-face feedback session (13). DOPS is scored evaluations of practical procedures and clinical examinations. This method of assessment has been shown to be valid, reliable and feasible in evaluating postgraduate medical registrars in the UK (14).

DOPS is considered as a valuable learning opportunity for trainees to enhance performance in a skill. Intricate working between trainee and assessor is required for its timely and effective functioning. DOPS assessments are tailor-made to be conveniently integrated into trainees’ and assessors’ daily routine and hence cconsidered highly feasible (15). A feed-back survey responded by 25 of the 27 pre-registration house officers completing the assessments showed that the majority (70%) was helped by direct observation in improving their clinical skills (5). However, a study conducted in 2009 showed that a number of interns agreed that DOPS might enhance their clinical acumen, but this evidence has not been reported scientifically, and the number of respondents was very small (16).

Discussion of clinical cases

Case-based discussion (CbD)

A CbD focuses entirely on the doctor’s real work and at all times explores exactly what was done and why and how any decision, investigation or intervention was decided upon (17). The trainee selects the timing, the records, and the assessor. A few days before the assessment, a case is chosen with particular curriculum objectives in mind and then discussed using focused questions designed to elicit responses that will indicate knowledge, skills, attitudes and behaviours relevant to those domains. After the discussion, the assessor rates the quality of the performance and then provides constructive feedback. On average, trainees are assessed six times during the year. A working plan for a typical CbD is as follows;

1. Planning

   • Trainee selects two medical records

   • Assessor selects a medical record for discussion and assessment

   • Trainee and assessor map out potential curriculum domains and specific competencies

   • Assessor prepares questions for discussion

2. Discussion

   • Assessor ensures that medical records are available during the discussion

   • Discussion starts with a reminder of medical record of the patient by assessor

   • Assessor explores the trainee’s clinical reasoning and professional judgment

   • Discussion is focused on the case, determining the trainee’s diagnostic and management skills

3. Feedback

   • Assessor provides effective and constructive feedback to the trainee

CbD evaluates what the trainees actually did rather than what they think they might do. This is the most striking difference between CbD and objective structured clinical examination (OSCE), which assesses the physician performance under examination conditions (18). CbD has been demonstrated to have significant face and content validity (19). In addition, it has been demonstrated that (with sufficient sampling) good levels of reliability (20) and validity with assessor training can be achieved (21). The innate nature of CbD demands that doctors’ own patients (cases) are used for a conversation or discussion that provides the main impetus to assess trainee’s applied knowledge, clinical reasoning and decision-making. CbD can explore a full range of holistic, balanced and justifiable decisions in complex situations, such as the ability to recognise dilemmas, managing a complex case in the given range of options, deciding on a course of action, explaining the course of action, and reflecting on the final outcomes.

Multisource feedback (360° assessment)

Mini-peer assessment tool (mPAT)

mPAT encompasses the integration of ideas about an trainee’s performance in a range of competence domains from their colleagues. This assessment strategy gathers confidential feedback from eight peers evaluating 16 aspects from the following domains (22);

• Diagnosis and appropriate application of available investigative tools

• Management of time

• Management of stress, fatigue, and workload

• Effective communication

• Knowledge of one’s own limitations

Foundation doctors in the UK are required to complete at least two mPATs per year. Archer et al explored the validity of mPAT by a mapping exercise against the UK Foundation Curriculum Trainees’ clinical performance (23). They administered a 16-item questionnaire written against a six- point scale on two separate events during the period of pilot study. The participants’ responses were evaluated to identify internal structural framework, potential points of leniency and various measurement variables. The analysis of these variables generated two main factors of clinical competence and humanistic values. The research illustrated that as a component of assessment program, mPAT has the potential to provide an effective and reliable tool of collating colleague opinions in comprehensively assessing the Foundation trainees.

Team assessment of behaviors (TAB)

TAB is a form of multisource feedback assessment for the trainee doctors in the UK Foundation Curriculum (24). TAB has following four domains based on the GMC’s guidance on professional behaviour;

• Developing and maintaining professional rapport and relationships with the patients

• Communicating by effective verbal skills

• Working in a team and as team leader

• Ensuring the accessibility and availability

TAB is used as a formative as well as a summative tool to help people improve their performances. This assessment tool entails a minimum of 10 returns for a valid, reliable evaluation. The recommended mix of raters is specified, since ratings vary significantly by staff group (25).

Patient satisfaction questionnaire (PSQ)

PSQ can provide formative feedback on a doctor’s professional performance within a process of appraisal (26). A structured questionnaire is used to obtain patients’ feedback. Physicians are expected to get feedback at least once every five years, to reflect on the feedback they obtain, and to use it to inform their further professional development, where appropriate (27). When patients assess physicians’ or larger health care systems, the demographics of the patients, and the questionnaire administration methods (e.g., postal, telephone or use of proxy respondents) can potentially influence final ratings (28) (29, 30). When colleagues judge the performance of other physicians, the rater’s personal impression, the duration and nature of the rater’s relationship with the examinee, and the rater’s familiarity with the doctor’s practice can jeopardise the entire assessment process (31).

Many reports in the existing literature suggest that multisource feedback can objectively assess key competencies like communications skills, interpersonal skills, collegiality, professional expertise, and the ability to progress in the medical field (21), (32), (33). Multisource feedback, however, has its own limitations. A number of studies have shown that responses tend to be skewed towards positive assessments of doctor performance (34) (35) (36) Others have shown dissatisfaction about the ability of multisource feedback, patient feedback in particular, in identifying the underperforming doctors (37).

Acute Care Assessment Tool (ACAT)

ACAT assesses the performance of a trainee after, for instance, a night shift in acute medicine

Strengths and weaknesses of WPBA

A study on the medical students indicated that WPBA was useful for increasing contact time with the supervisors (38). In another research, students reported that allocating a tutor in WPBA was the most effective means of judging the competence (39). By WPBA, learners establish professional relationship with their tutors, who become authentic sources of effective feedback for the learners. The outright strength of WPBA is its formative potential for assessment. The essential impetus needed to achieve ‘assessment for learning’ is the provision of feedback by the assessor, enabling the trainee to steer his or her learning towards the intended learning achievements (40). Evidence-based research has shown that systematic feedback delivered by a credible source can enhance clinical performance (41). The most striking evidence of performance improvement by WPBA is derived from studies exploring the feasibility and validity of the multisource feedback (5). Research from psychology literature has shown that multisource feedback can lead to gradual enhancements in professional competence (42), Another study of medical education showed that doctors getting specific feedback from peers, colleagues and patients can use the data to implement modifications in their clinical practice (43). The direct observation of trainee’s performance at the workplace is only made useful by the associated feedback, thus triggering reflection (44). The feasibility of WPBA is amplified than the conventional assessments as it is applied and conducted during the course of day-to-day routine. Although WPBA demands prior training of the faculty, additional time and student knowledge and sensitisation, institutions don’t need any dedicated infrastructure to establish this assessment strategy. However, a survey reporting the views of 539 surgical trainees on the Intercollegiate Surgical Curriculum Program(16) showed that 60% of respondents felt that the program adversely affected on training opportunities due to a long time required to complete the assessments. More than 90% stated that the program had a neutral or negative impact on their training overall.

WPBA, per se, cannot replace traditional methods of assessment but carries a potential of add-on method especially to the in-training or formative assessment. The trainees who perform well in initial encounters may get overconfident and this may be a hurdle in motivating them for future improvements (44). Since WPBA demands a lot of time, trainees tend to seek less senior assessors. There is evidence to suggest that the more senior and expert staff may provide lower but more accurate rating of performance (45). At the same time, WPBA requires assessor training particularly in objective evaluations and providing effective feedback. Sensitization and introductory seminars about WPBA may be the first step in grooming the staff about appropriate functionality of this assessment tool.

Conclusion

WPBA involves evaluation of performance and provision of feedback of doctors in their everyday activities. These assessment tools purport to provide valuable insight to trainee, assessor and academics, and are found to have some educational impact on learning. However, further evidence-based interventional and experimental models are needed to establish its significant educational impact in medical education.