Skip to main content
NCBI home page
The BMJ logoLink to The BMJ
. 2001 Jun 30;322(7302):1600.

Assessing clinical competence and revalidation of clinicians

Simulated surgeries would have advantages

Peter Burrows 1
PMCID: PMC1120634  PMID: 11458897

Editor—McKinley et al present cogent arguments for directly assessing the consultation competence of general practitioners undergoing revalidation.1 They identify the required attributes of an assessment as reliability, validity, acceptability, feasibility, and educational impact. They do not mention, however, that the methodology must also be capable of setting a consistent standard for pass/fail decisions for every candidate.

The model that they offer depends on the judgment of two observers watching 10 consecutive consultations, a task that they might undertake 12 times in a year. If the prevalence of insufficient competence was 2% they might encounter it once every four years. How could they apply a minimum standard reliably, along with their 1000 assessor colleagues? Just 10 randomly presenting cases will be too few to guarantee an adequate coverage of consultation skills, and in the absence of standardisation it will be hard to achieve a reliability score above 0.5. Furthermore, the patients will behave differently if they know that their doctor is under assessment, thus compromising validity.

The 12 case simulated surgery for the MRCGP (membership of the Royal College of General Practitioners) examination is a model that would overcome these difficulties. The cases are written to test specific skills; the patients are role players who are trained to present the same challenge to each candidate; each examiner observes the same case throughout the circuit and marks the candidate's performance on a predetermined schedule; each candidate is assessed by all 12 examiners, which minimises bias; and the pass mark is set by the aggregation of the judgments of all the examiners, based on the cases they have marked.

This examination has yielded reliability figures (coefficient α) in excess of 0.8 in all six administrations over three years. Face validity is high (“like a locum surgery”), and content validity is controlled. The method is acceptable (universally preferred to video by eligible candidates). The cost is less than £350 per candidate if the circuits are full. A profile of skill subscores can be presented to the candidate, providing formative feedback. If the administration was put in place, regular simulated surgeries could be offered for revalidation in regional venues throughout the United Kingdom.

References

  • 1.McKinley RK, Fraser RC, Baker R. Model for directly assessing and improving clinical competence and performance in revalidation of clinicians. BMJ. 2001;322:712–715. doi: 10.1136/bmj.322.7288.712. . (24 March.) [DOI] [PMC free article] [PubMed] [Google Scholar]
BMJ. 2001 Jun 30;322(7302):1600.

Video assessments might be effective screening method

Nick Gravestock 1

Editor—McKinley et al have shown the reliability and feasibility of their consultation assessment for examining consultations in general practice, but they have glossed over some issues.1-1

The first such issue is the desirability of an assessment that tries to be both formative and summative. Trying to do both at the same time is generally accepted to be problematic.1-2 Although an assessment method may be valid when used in both ways, the stress of undertaking a summative assessment may affect performance adversely. The idea that the five attributes of assessment are multiplicative is not a true mathematical concept, and the absence of one does not negate the others. A summative assessment does not have to have educational impact after the assessment: its impact is in setting the learning aims and objectives for learning before assessment.

The second issue is that of patient acceptability. The authors raise the problem that certain types of patients decline to be videoed, but it might be expected that the same patients will decline to have two strangers sitting in the consultation. Videoing can be intrusive, but around three quarters of patients forget the camera during the consultation.1-3 It is far harder to forget two extra people.

The third issue is reliability. The table on bmj.com shows high figures for the validity, but the number of observers who were involved is not clear. Increasing the number of observers is likely to decrease the interobserver reliability.

The summative assessment video component may only have inter-rater reliability that is “not impressive,” but its reliability in screening poor performers is reckoned to be 95%, these then undergoing further assessment.1-4 If the purpose of the exercise is to select out underperforming general practitioners then a similar tool would be effective, and possibly cheaper as a screening method. The problem of finding enough assessors remains.

References

  • 1-1.McKinley RK, Fraser RC, Baker R. Model for directly assessing and improving clinical competence and performance in revalidation of clinicians. BMJ. 2001;322:712–715. doi: 10.1136/bmj.322.7288.712. . (24 March.) [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 1-2.Brown G, Bull J, Pendlebury M. Assessing student learning in higher education. London: Routledge; 1997. p. 251. [Google Scholar]
  • 1-3.Martin E, Martin PML. The reactions of patients to a video camera in the consulting room. J R Coll Gen Pract. 1985;34:607–610. [PMC free article] [PubMed] [Google Scholar]
  • 1-4.Campbell LM, Howie JGR, Murray TS. Use of videotaped consultation in summative assessment of trainees in general practice. Br J Gen Pract. 1995;45:137–141. [PMC free article] [PubMed] [Google Scholar]
BMJ. 2001 Jun 30;322(7302):1600.

Authors' reply

Robert K McKinley 1,2, Robin C Fraser 1,2, Richard Baker 1,2

Editor—Burrows advocates the simulated surgery used during the MRCGP (membership of the Royal College of General Practitioners) examination as a potential tool for assessing consultation competence for revalidation.2-1 Although this approach has some commendable features (for example, a guaranteed range of clinical challenges), we have previously questioned its credibility, let alone validity.2-2

Burrows admits that the simulated surgery for the MRCGP examination does not test ability to make diagnoses or to elicit abnormal physical signs. Surely both are fundamental skills for all clinicians. The marking system is also peer referenced and not criterion referenced, which is essential for equity in regulatory assessments.

We can reassure both Burrows and Gravestock about the reliability that can be achieved with multiple assessors. We have reported a study of the assessment of the consultation competence of 180 medical students involving 86 trained assessors, most of whom assessed fewer than five students. The interassessor reliability of pass/fail decisions was 0.94.2-3

This high level has been replicated in regulatory assessments of over 140 general practice registrars in Kuwait, again involving multiple pairs of assessors, both internal and external (paper in preparation). With minimal screening, 10 patients provided sufficient clinical challenges. These data confirm that assessments such as we propose can be highly reliable when performed by multiple assessors.

Gravestock asserts that summative assessment need not have educational impact and, consequently, that the five essential attributes of an assessment instrument are not truly multiplicative. All candidates can benefit from explicit feedback on how to improve, and those who fail have a right to be informed why the assessors reached their judgment. This is particularly essential when the livelihood of a professional colleague is at stake. It is widely accepted by leading medical educationalists that the essential attributes are indeed multiplicative.2-4 Incidentally, all assessments can be stressful irrespective of methodology—this is a fact of life.

We agree with Gravestock that patient acceptability is important. Indeed, we emphasised this in our paper. Over 8000 patients have participated in our assessment studies in Leicestershire,2-3 Kuwait, and Hong Kong, and all these involved the presence of two assessors in the consulting room. The circumstances are fully explained to patients, whose consent is mandatory. There have been few refusals and no adverse comment by the patients.

The authors of both these letters support our contention that assessing clinical competence through direct observation must be central to the revalidation of clinicians. Like them, we do not underestimate the logistical implications. Nevertheless, the profession needs to get on with the task.

References

  • 2-1.Burrows PJ, Bingham L. The simulated surgery—an alternative to videotape submission for the consulting skills component of the MRCGP examination: the first year's experience. Br J Gen Pract. 1999;49:269–272. [PMC free article] [PubMed] [Google Scholar]
  • 2-2.Fraser RC, McKinley RK. Simulated surgery. Br J Gen Pract. 1999;49:753. [PMC free article] [PubMed] [Google Scholar]
  • 2-3.McKinley RK, Fraser RC, van der Vleuten C, Hastings AM. Formative assessment of the consultation performance of medical students in the setting of general practice using a modified version of the Leicester assessment package. Med Educ. 2000;34:573–579. doi: 10.1046/j.1365-2923.2000.00490.x. [DOI] [PubMed] [Google Scholar]
  • 2-4.Van der Vleuten CPM. The assessment of professional competence: developments, research and practical implications. Advances in Health Sciences Education. 1996;1:41–67. doi: 10.1007/BF00596229. [DOI] [PubMed] [Google Scholar]
BMJ. 2001 Jun 30;322(7302):1600.

Results from pilot study using portfolio and 360° questionnaire

Robert Mason 1,2, Louisa Zouita 1,2, Brian Ayers 1,2

Editor—Regular appraisal of consultants is to become an integral part of the new NHS.3-1 McKinley et al addressed the problems inherent in the process.3-2 We present the initial results of a pilot study of appraisal using a portfolio and incorporating a 360° approach.

The project included 20 consultants, who completed a portfolio based on the General Medical Council's guidelines for revalidation.3-3 A 360° appraisal questionnaire was distributed by each consultant to 10 team members and 20 patients; it was based on the American Board of Internal Medicine validated questionnaire and had questions on attitude, communication, and skills.3-3 The return rate for patients was 50% and that for team members 84%.

No problems were reported with completing the portfolio questionnaire, and consultants had no problem with job plans, including both NHS and independent practice. The only sections impossible to complete concerned critical incidents and complaints. Currently, this information is not collected centrally in our trust. The median score for all consultants for both the team and patients was in the very good or excellent category. This was reassuring. Participants thought, however, that the distribution of questionnaires to patients and team members should be random and handled centrally. The anonymity of replies was considered to be crucial.

The broadest spread of replies came from clinicians with primary care responsibilities rather than those in service specialties. Scoring was consistent for individual clinicians, with the same spread seen in all questions. Comments from consultants shown the overall results of team and patient questionnaires varied from complimentary to outright rejection.

The time taken to undertake this pilot study in only 20 consultants was 40 hours and 2000 pieces of paper. A large institution with 500 consultants would need 1000 hours and 50 000 pieces of paper, as well as 2500 hours to conduct appraisal (a minimum of 5 hours per consultant, including preparation)—716 consultant sessions.

We conclude therefore that appraisal by portfolio and 360° assessment is feasible if properly funded. Further work is necessary to simplify the assessment of team and patient perception. Emphasis must be placed on minimising the time spent to reduce the impact on service delivery. If the process is handled sympathetically and with common sense, however, it will be a benefit to all.

References

  • 3-1.NHS Plan. A plan for investment. A plan for reform. www.nhs.uk/nhsplan (accessed 21 June 2001).
  • 3-2.McKinley RK, Fraser RC, Baker R. Model for directly assessing and improving clinical competence and performance in revalidation of clinicians. BMJ. 2001;322:712–715. doi: 10.1136/bmj.322.7288.712. . (24 March.) [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3-3.General Medical Council. Revalidation: the regular demonstration by doctors that they remain fit to parctise. www.gmc-uk.org/revalidation (accessed 21 June 2001).

Articles from BMJ : British Medical Journal are provided here courtesy of BMJ Publishing Group

RESOURCES