Workplace assessment for licensing in general practice

Abstract

With the redesign of general practice training implicit in the Department of Health's programme of reform, Modernising Medical Careers, there is the opportunity to bring summative assessment and the MRCGP examination together into a unified assessment framework for licensing. It is likely that assessment in the workplace will play a central role in such a process. Workplace assessment is of high validity and has the potential to reconnect teaching and testing in general practice. Five principles to underpin the design of a workplace assessment are proposed, namely that it should be: competency-based, developmental, evidential, locally assessed, and triangulated. Successful implementation of workplace assessment will not only serve to reduce the current testing burden on trainees, but will also harness the involvement of medical teachers. In doing so, general practice has the opportunity to create a powerful tool for professional development.

INTRODUCTION

The assessment of the competence of doctors by examinations is inherently problematic. There are real differences between what doctors do in controlled assessment situations and their actual performance in professional practice,¹ and degrees of correlation between the two have been shown to be extremely variable.² Competence is a prerequisite for performance, but no matter whether the assessment takes place in a controlled or uncontrolled setting, competence can only ever really be inferred from performance. In other words, competence indicates what people can do in a contextual vacuum, under perfect conditions, whereas performance is about how people behave in real life, on a day-to-day basis.²

Recent policy developments have led to a fundamental review of the assessment of doctors in training, with a growing emphasis on competency-based training and assessment. Modernising Medical Careers³ recommends that all examinations of medical training now be competency-based and premised on the General Medical Council's (GMC) document Good Medical Practice.⁴ Overseeing the implementation of this policy is the Postgraduate Medical Education and Training Board (PMETB), which is shortly to assume the functions of the Joint Committee of Postgraduate Training for General Practice and the Specialist Training Authority. PMETB has issued guidance for the development of trainee assessment in all specialties. In its published principles and standards for an assessment system for postgraduate medical training,⁵ PMETB defines a satisfactory assessment system as:

‘… an integrated set of assessments which is in place for the entire postgraduate training programme, and which supports the curriculum. It may comprise different methods, and be implemented either as national examinations, or as assessments in the workplace. The balance between these two approaches principally relates to the relationship between competence and performance. Competence (can do) is necessary but not sufficient for performance (does do), and as experience increases so performance-based assessment becomes more important.’⁵

It is clear from statements issued by the Department of Health and PMETB that the Royal Colleges will play a central role in defining the requirements of postgraduate training. In keeping, therefore, with the other medical colleges, the Royal College of General Practitioners (RCGP) set about developing a curriculum for general practice training that is expected to be delivered in 2005. In many ways, the RCGP was ahead of the game and a comprehensive syllabus for the existing MRCGP examination was already in existence, having been drawn up after widespread consultation with all stakeholders in 2002.⁶ The MRCGP syllabus, Good Medical Practice, and the emergent curriculum statements may usefully be brought together in an assessment matrix — a framework from which to derive competences for general practice and how best these might be tested. Good Medical Practice also forms the basis of evidence required to be collected for the NHS Appraisal for Doctors in Training framework.⁷ In order to reduce the assessment, and evidential burden on trainees, any proposed system of assessment should also satisfy the data collection requirements of the NHS Appraisal framework.

How this fits in

The assessment of doctors training for general practice should be reviewed as a result of growing evidence from the educational and assessment literature. Workplace assessment offers ‘authenticity’, focusing on what doctors actually do in the workplace, and should therefore form an integral part of any assessment framework for assessing doctors. The principles underpinning workplace assessment are that it should be: competency based, developmental, evidential, locally assessed and triangulated with other assessments.

From both a political and an educational perspective, the time would then appear to be right to rethink the assessment of doctors in training for general practice. In this article we argue for a central role for workplace assessment as part of a global licensing framework for doctors in training. Furthermore, we wish to establish a number of key features, which we believe should underpin the development of such an assessment in the workplace.

WORK-BASED OR WORKPLACE?

Work-based assessment can be considered in a number of ways and may include assessments of work undertaken ‘off-site’, such as the current MRCGP video module, surrogate tests of performance routed in work, simulated surgeries, and assessments of outcomes, process of care and practice volume that are undertaken in the working environment.⁸ A more specific term, which will be used throughout this paper, is ‘workplace assessment’, that is, assessment of working practices undertaken in the working environment.

WHY USE A WORKPLACE ASSESSMENT?

There are two main arguments for including workplace assessment in a licensing examination for general practice. The first of these is educational and concerns the re-coupling of teaching and testing. Assessment should be an integral part of the curriculum, not a ‘bolt-on’ extra at the end. Trainees (and indeed trainers) should know exactly what is expected of them, and have the opportunity to demonstrate attainment over time and in a variety of contexts.⁹

A second argument, well supported by a wealth of research material, suggests that an assessment is more valid the closer it gets to the activity one wishes to assess.^10,11 If, for example, you want to know how a doctor consults, watch him do it. In the lexicon of test design, this is called ‘authenticity’. Authenticity is particularly important when dealing with the assessment of competence because expertise appears to be domain-specific and contextual.¹² That is, good communication skills in the consulting room may not be transferable to, say, an oral examination. Furthermore, some competency areas such as probity, availability and professionalism, simply cannot be disentangled from system (for example, practice facilities) or personal influences (for example, health). As such, assessment of performance, as opposed to competence, provides us with the only route into many of the areas that we might wish to assess. Context, therefore, is highly significant in both the development and expression of competence and, as such, has an important part to play in both learning and assessment.

One way to increase the authenticity of an assessment method is to base it on the simulation of reality, which enhances reliability through standardisation. But this ‘preoccupation with objective testing encourages the substitution of surrogate or indirect measures for the real thing’¹³ and has dominated thinking in medical education, particularly in relation to high stakes assessment. Interestingly, as Schuwirth¹⁴ highlights, test development is coming full circle from patient-based examinations through simulations back to observations of real practice.

LESSONS FROM EXISTING ASSESSMENT METHODS

Summative assessment in the UK already includes a workplace assessment in the form of the Structured Trainer's Report; it could, however, be argued that this is not an assessment tool as such, but rather a portfolio of smaller assessment units. The Structured Trainer's Report has stood the profession in good stead but there are issues that arise from it as it stands.

• The Report specifies failing criteria but provides no developmental markers as to how the learner is progressing. As such, its educational impact is limited.

• The content of the Report includes a large section on basic clinical skills which lend themselves to more rigorous testing in a standardised way, as is done when a doctor's license is brought into question, such as in the GMC's performance procedures.¹⁵ It could also be argued that such skills should be tested before entering basic specialist training rather than at the end of it.

• It is rare to fail summative assessment on the Trainer's Report alone¹⁶ and when this does occur, supporting documentation is usually required from the trainer indicating that there is uncaptured information about the registrar's inadequacies that is not stipulated in the Report.

• Despite recommendations for its use as an open document, anecdotal reports from directors of general practice education suggest that the Trainer's Report often remains untouched until the last few weeks of training.

• The Structured Trainer's Report is ‘moderated’ in theory, as it has to be approved and countersigned by a director of postgraduate general practice. This, however, is rarely more than a rubber-stamping exercise and, unlike the other components of summative assessment, there are no national attempts at calibration or standardisation.

OUTLINE PROPOSAL

Our proposal is that any national workplace assessment for licensing should have a number of key features (Box 1). Furthermore, the development of the assessment will need to take into account validity, reliability, feasibility, cost and educational impact. These issues are given individual consideration in the next section of this paper.

Box 1. Key features of the national workplace assessment for licensing.

• ▸Competency-based
• ▸Developmental
• ▸Evidential
• ▸Locally assessed
• ▸Triangulated

Competency-based

Competency-based training and education has acquired a bad name. Criticisms of competency movement abound^12,17-19 accusing it of being overly simplistic, atomistic and reductionist. In spite of that, ‘in graduate medical training in the UK a competency model is being promoted with an almost messianic fervour’.²⁰ There exists a general confusion between competence and competency, and idiosyncratically constructed competency frameworks are springing up everywhere.^21-23 Despite these criticisms, Gonczi¹² argues that a competency-based approach to education and training can be made to work and is as applicable to the professions as to any other occupation. He cautioned, however, that there are a number of ways of conceptualising competence and that ‘if the inappropriate way is adopted, not only will this potential fail to be realised but serious damage will be done to skill formation policies in the medium term’. Gonczi's three conceptualisations of competency can be summarised as:

• task-based

• generic attributes

• holistically defined in context.

The task-based, or behaviourist, competency model is the one usually adopted when competency-based training and assessment is being proposed. As Rethans states, ‘competency-based assessment measures what doctors can do in controlled representations of professional practice’.¹ This brand of competency-based training and assessment focuses on discrete behaviours associated with the completion of atomised tasks. Indeed, this is its appeal in that the model is both simple and clear. However, such an approach to education, training, and assessment is generally agreed to be conservative and reductionist, ignoring as it does, underlying attributes, group processes, context, the complexity of performance, and the role of professional judgement. Gonczi sums up the view of many other authors before and since in declaring that such a model is ‘clearly inappropriate for conceptualising professional work’.¹² Leung and Diwaker, in a recent critique of competency-based training, concur and caution that ‘we should be cautious of applying the competency-based approach universally unless robust higher order competencies are available’.¹⁹

An alternative approach — that of treating competencies as general personal attributes — is popular in the management field and can be found, for example, in the leadership literature, such as that of Goleman.²⁴ We know, however, that expertise is context specific and general attributes may not be applicable in certain circumstances. General attributes are therefore unhelpful in the design of training programmes or for the purposes of wider accountability.

Modernising Medical Careers³ recommends that all postgraduate programmes should be ‘competency-based’. Modernising Medical Careers: The Next Steps²⁵ — although ostensibly focusing on the Foundation programme — softens this considerably:

‘MMC[Modernising Medical Careers] signalled a move to competency-based training throughout the medical continuum, which will be reflected in Foundation programmes. Evolving thinking takes this a step further and suggests progress should be outcome-based — that is, not just acquisition of competencies but a demonstration that they can be applied in real situations’.²⁶

This later statement fits with the idea of competencies as general attributes within a context, incorporating both understanding and judgement: that is, competence ‘as a complex structuring of attributes needed for intelligent performance in specific situations’.¹² We should also note here the shift in discourse as the concept of ‘intelligent performance’ comes from an integrated approach to constructing competencies and a move towards the more holistic construct of competence based on outcomes defined by a national curriculum.

Good Medical Practice for General Practitioners²⁷ provides us with units of competence, defining the characteristics of the excellent (and unacceptable) GP. From these, meaningful sub-divisions or ‘elements’ can be defined. Performance criteria can then be developed, describing the sort of behaviours that might lead one to infer that competence has been obtained. It will, of course, be essential that the workplace assessment ultimately fits with the RCGP's curriculum which, interestingly, takes the agreed European definition of family practice²⁸ as its point of departure.

Developmental

The Structured Trainer's Report is a portfolio of assessments. Although there is little evidence to support the use of portfolio assessments for summative assessment in medical education,²⁹ they have been widely used in other fields and in other countries.³⁰ Any new workplace assessment for licensing offers the opportunity to link training and assessment more effectively; this proposal recommends the development and testing of a ‘developmental assessment’ that is a ‘process of monitoring student's progress through an area of learning so that decisions can be made about the best ways to facilitate future learning’.³¹ This provides the opportunity for ‘scaffolding’,³² a process of support and guidance that can be offered to enable the trainee to achieve at a potentially higher level.

This may appear to mix formative and summative assessment but as Sadler³³ points out, the methods are often the same, the only difference being the timing of application. Successful examples exist: the Royal Australian College of General Practitioners recently considered in-training assessment (ITA) as part of its training programme and Hays and Wellard³⁴ described the possibility of introducing some summative assessment tasks during training in a way that complements rather than interferes with formative assessment processes. Prescott³⁵ also described a formative assessment tool for dental trainees in which information gathered formatively is used qualitatively rather than quantitatively towards a summative decision. The aim of any workplace assessment must be explicit³⁶ but the use of a developmental portfolio — where students are guided through progressions of well described criteria towards specified goals — informed by a predefined ‘sufficiency of evidence’ may be one way of reconciling formative with summative purposes.

Miller³⁷ provided us with a simple, and now familiar, model for the development of clinical competence (Figure 1) and it should be possible to build such a progression into a workplace assessment, defining performance criteria at each level of Miller's pyramid. There are, of course, other such developmental progressions in the literature, such as that described by Dreyfus and Dreyfus.³⁸ Interestingly, the latter has already been adopted for use in GP training in a national out-of-hours training workbook.³⁹ By drawing up a continuum of competence, an explicit structure for development is provided, competencies can be demonstrated ‘when ready’, and weaknesses exposed can be worked on early in training.

Figure 1.

Miller's Pyramid.³⁷

Evidential

A national workplace assessment for licensing would be dependent on the collection of evidence. Public sector workplace assessments in Australia⁴⁰ point the way. A national body would be responsible for developing and maintaining the competency framework for general practice training and also issue guidance on what would constitute ‘sufficiency of evidence’ or ‘standards of proof’. This body could also develop, or be the repository for, a range of approved in-service assessment tools. A range of modes of workplace assessments (such as observation, 360-degree appraisal, case analysis) would be encouraged.

An overall holistic assessment of the evidence presented, perhaps by an external examiner, needs also to be considered and might serve to aid comparability across sites. In the UK, external moderation could be carried out by a trained national panel of assessors.

In their analysis of the relationship between summative and formative assessment, William and Black⁴¹ recommended ‘separating the interpretation of evidence from its elicitation, and the consequent actions from the interpretations’. This supports the concept of ongoing evidence collection throughout the training period but with regular, well circumscribed staging reviews at which the developmental framework is reviewed and the learner's progress judged.

Locally assessed

Workplace assessment for licensing would be carried at local level predominantly by the GP trainer. This raises the thorny issue of reliability. As Southgate¹⁵ points out, ‘establishing the reliability of assessments of performance in the workplace is difficult because they rely on expert judgements of unstandardised material’. In workplace assessment, as with any other form of assessment, there are several potential threats to reliability:⁴² inter-observer variation (the tendency for one observer to mark consistently higher or lower than another), intra-observer variation (the variation in an observer's performance for no apparent reason — the ‘good/bad day’ phenomenon) and case specificity (the variation in the candidate's performance from one challenge to another, even when he/she seems to test the same attribute).

Despite these challenges we should not eschew a workplace assessment as, although reliability in performance assessments can be elusive, it can also be maximised through a series of measures outlined by Baker et al:⁴³

• specification— of standards, criteria, and scoring guides

• calibration — of assessors and moderators

• moderation — of results, particularly those on the borderline

• training— of assessors, with retraining where necessary

• verification and audit— through quality assurance measures and the collection of reliability data.

A similar list in relation to portfolio assessment is suggested by Klenowski.³⁰

Reliability will always be a concern in workplace assessment,⁴⁴ but it can be enhanced. Gipps⁴⁵ took this further in arguing for a new paradigm of ‘educational assessment’ and suggested dropping the term ‘reliability’ completely when considering performance assessments and replacing it with ‘comparability’ based on ‘consistency’. Assessment is not an exact science and, despite the inherent challenges, workplace assessment is a paradigm worth pursuing. No other form of testing is as direct, relevant and capable of measuring holistically the higher integrative functions that make up professional competence.

Clearly, the introduction of a national workplace assessment will require a complementary training programme, arrangements for calibration, a procedure for the moderation of results and a raft of quality control and reliability checks. But it will be worth the effort. The more that teachers can be engaged in assessment — in selecting methodologies, generating standards and discussing criteria — the more they will be empowered in the educative process. To ignore this professional need, and to impose another round of externally generated assessments, will serve to disaffect further the teaching community. Of course, this assumes that GP teachers will want to be involved in assessment at all; if workplace assessment is to succeed in its purpose, there are some complex issues to work through concerning the professionalisation of our teachers and the impact of bringing assessment closer to home on their relationship their learners.

Triangulation

Through triangulation within workplace assessment we can be more confident of the veracity, fairness and reproducibility of our final judgement. Triangulation, both over time and context, is important because of the variability of those situations and contexts in which learners will find themselves. In this way, performance assessment moves us from the quantitative paradigm of psychometrics to the rich description of qualitative research. In view of the complexity of professional performance, there is also the need to triangulate what is found in the workplace with other assessments. This approach is wholeheartedly endorsed by social science researchers and the assessment literature, the latter arguing for a realistic mixture of decontextualised structured exercises and contextualised performance tasks.⁴⁶ Again, the GMC's performance procedures provide a model whereby workplace assessments are supported by ‘phase two’ tests of ‘knowledge’ and ‘skills for clinical method’.¹⁵

In view of the need for triangulation, it is vital that licensing assessments are coordinated and that a national body maintains an overview. Such a body both needs to consider the emergent assessments of the Foundation Programme of Modernising Medical Careers,³ and to ensure that all domains within the competency matrix are assessed, that they are assessed adequately, and that they are assessed at the appropriate stage of training.

DISCUSSION

The implementation of a workplace assessment for GP licensing, although politically driven, is underpinned by a sound rationale supported by the theoretical literature. Assessments conducted in the workplace are of high validity and serve to reconnect teaching and assessment. A competency-based model accords with the overall contemporary emphasis of medical assessment; caution, however, is advised lest defined competencies become over-atomised. In order to enhance educational impact, the use of holistic competencies within a developmental continuum is recommended. Such a continuum has the advantage of illustrating explicitly the direction of travel for trainees rather than merely pointing out the level below which they should not fall. In this we are entirely in agreement with Eraut:⁴⁷

‘At the very least, we recommend that assessment systems should be capable of recording achievement beyond competence … Such systems should also be coherent, though not necessarily identical with those being developed for recording continuing professional development’.

To strengthen further the link between teaching and assessment, and to deal with the practical expediencies of large-scale implementation, a workplace assessment should be locally assessed and based on the collection of evidence. A sufficiency of evidence would be predefined and triangulation built in as an essential feature in order to enhance the reliability of judgements made.

Clearly, there is much to be done in the development of a workplace assessment for licensing to create a vehicle for assessment that is robust, fair, comparable and consistent; further research in this area is urgently required. To get it right will not only reduce the current assessment burden on trainees, but also harness the involvement of medical teachers. In doing so, we have the opportunity to create a powerful tool for professional development.

Competing interests

None