Undertaking Procedure-Based Assessment is Feasible in Clinical Practice

Abstract

INTRODUCTION

With the development of a new curriculum, workplace based assessments such as procedure-based assessment (PBA) are becoming increasingly common within surgical training. However, there have been concerns about the impact of these assessments on clinical practice. This study assessed the time taken to complete PBA forms to determine whether it is feasible in clinical practice.

MATERIALS AND METHODS

PBAs for three colorectal procedures (anterior resection, right hemicolectomy and anal fistula) were undertaken by various trainers and trainees. A pilot study was performed to identify potential reasons for incomplete forms and procedural modifications subsequently applied in the main study. Times taken to complete the consenting and operative components of the forms were recorded.

RESULTS

Incomplete forms in the pilot were mainly attributable to time constraints. In the main study, all assessments were completed within 30 min. Assessment times increased with complexity of the procedure. Median times for completing the consenting and operative components in anterior resection were 13 min (range, 8–15 min) and 15 min (range, 10–18 min), respectively.

CONCLUSIONS

PBAs are feasible in clinical practice and are valued by trainees as a means of enabling focused feedback and targeted training. Commitment from trainers and trainees will be required but, with adequate planning, the assessment tool is effective with minimal impact on clinical practice.

Medical training in the UK is undergoing significant change. The publication of Modernising Medical Careers¹ in 2003 has resulted in a new framework for training in medicine with an emphasis on competency-based progression. The Postgraduate Medical Education and Training Board (PMETB) has been established and now assumed its statutory powers. It has stipulated principles that should underpin assessment systems in all postgraduate medical training² and workplace-based assessment represents one means through which this can be achieved.³ A variety of workplace-based assessments, including the procedure-based assessment (PBA), will be utilised in surgical specialities⁴ and will contribute both formative and summative roles.

The Intercollegiate Surgical Curriculum Project (ISCP)⁵ has been developed in response to current issues within surgical training and has, at its centre, a revised curriculum through which it aims to deliver first-class postgraduate surgical training. The ISCP has generated PBAs for the index procedures in each speciality. Within coloproctology there are PBAs for anterior resection, right hemicolectomy and surgical management of a low anal fistula and, more recently, closure of ileostomy has been added. Each PBA involves a trainer directly observing a trainee's surgical skills and then completing a form in which a range of competencies is assessed. These competencies are divided into six core domains which include the consenting procedure, intra-operative technique and postoperative management.⁵

Concerns have been raised that the forms are detailed and that completion may have a deleterious effect on clinical practice at a time when there is increasing time pressure to meet NHS targets. The purpose of this study was to assess the time required to complete PBAs and their ease of use in the surgical workplace.

Materials and Methods

The study was undertaken within the colorectal department of a district general hospital comprising three consultants and two specialist registrars (years 2 and 6). PBAs were undertaken for three colorectal index procedures (anterior resection, right hemicolectomy and fistula in ano). The forms were split into two components – the consenting procedure and the remainder under the heading ‘operative component’. An initial pilot study was undertaken on seven cases. Based on the outcome of this, various changes were made for the main study which comprised 15 cases, five for each index procedure. The consent and operative sections were completed by various combinations of trainer and trainee.

There are three components to completing PBAs – observation of the consent/procedure, giving feedback and completing the form. In the second half of the study, feedback was given whilst the form was completed and we found the form to be a useful aide memoire to enhance the feedback. For the consent process, feedback and completion of the form took place immediately after the consultant had observed process. The total time taken to observe consent and provide feedback with the form was recorded. For the operative section, observation time (length of operation) was not recorded but the times taken to provide feedback with the form were noted.

Results

Pilot study

None of the seven PBA forms was fully completed in the pilot study. Reasons for this were time constraints of the trainee (2 forms), time constraints of the trainer (4 forms) and unsuitability of case which was not predictable pre-operatively (1 form). In the latter, the patient underwent a Hartmann's procedure rather than anterior resection.

Main study

For the main study, appropriate cases were selected in advance. Time slots for observing the consenting procedure were identified in advance and time was also allocated directly after each case in order to undertake the operative assessment component.

Median times taken to complete each half of the assessments are shown in Table 1. As expected, greater time was required for the more complicated procedures. However, both components of each of the three PBAs were completed in less than 30 min.

Table 1.

Median times taken to complete each component of the procedure-based assessment

Procedure	Consent assessment (min)	Operative assessment (min)
Low anal fistula	5 (4–7)	6 (4–8)
Right hemicolectomy	7 (6–8)	13 (8–15)
Anterior resection	13 (8–15)	15 (10–18)

Median values with range in parentheses.

Completion of the PBAs resulted in focused feedback to the trainees about their practice. As a result, the trainees in this study valued this structured approach because it enabled subsequent training to be targeted appropriately.

Discussion

The pilot study highlighted potential pitfalls when introducing PBAs to surgical training. In order to achieve properly completed assessments, it was necessary to modify our practice significantly. This included identifying appropriate cases pre-operatively and allocating sufficient time immediately after the operation for completion of the assessment. Therefore, commitment from both trainers and trainees is necessary both in terms of absolute time involved and of educational attitude towards training if maximum benefit is to be gained from PBAs.

Within the main study, times taken for assessments were deemed practical in the working environment. The times taken to complete the assessments were broadly in line with the ISCP prediction of 20 min. Dividing the assessments into consent and operative components, as in this study, was useful as it permitted the separate components to be completed on different occasions. The consent and operative components do not necessarily need to be completed on the same patient. This modification is another measure that minimises the perceived time burden of completing the assessments whilst retaining their significant educational benefit. As a result, it was possible to complete the operative component of an assessment between cases on a theatre list. Consultants may not be used to allocating time to observe the consent process but, with appropriate planning, observation, feedback and completion of the form, was possible without impacting on service commitments.

The formative assessment function of PBAs was well fulfilled because the trainees valued the focused feedback that occurred and subsequent targeted training that was put in place. By structuring the feedback around the PBA forms, the consultants felt there was a benefit to the process rather than another form–filling exercise. However, the trainees felt the need to develop and respond to the targets that had been highlighted and, therefore, we would recommend using the PBAs at intervals throughout the course of a training placement rather than, as has been suggested, every time an index procedure is undertaken.⁴ The alternative recommendation of a minimum of six PBAs per annum for each index case is perhaps more realistic⁵ and would allow development of skills between each assessment. In turn, this prevents the assessment from becoming too repetitive and allows new, realistic targets to be agreed between the trainer and trainee after each evaluation. The disadvantage of setting a minimum requirement is that trainees regard the minimum as the norm and, therefore, practice informally simply to achieve better scores. This defeats the purpose of an objective assessment method that is designed to aid learning.

Conclusions

Use of PBAs will become commonplace in surgical training. They offer great potential as an educational tool but, to gain maximum benefit, there will need to be commitment from both trainers and trainees. With appropriate planning and identification of suitable cases, PBAs can be used to provide a basis for structured feedback for trainees that will allow planning of subsequent training for the individual trainee. The time taken to complete assessments is short and has minimal impact of delivery of surgical services.