EVALUATING THE EVALUATORS IN POSTGRADUATE FELLOWSHIP EXAMINATIONS - A VIEWPOINT

Abstract

When the result of a medical examination for a group of candidates is bad, it may be because the candidates are bad; or the questions and the format of the examination are improper. The reason may also be because there is no teaching and preparation of the residents or the mode of examination and the methods of assessment are flawed. This paper tries to focus attention on the role of the examiners in conducting a medical postgraduate examination. It highlights the negative impact of the aggressiveness of the evaluators, the unnecessary interruption of the candidate by the examiners and flawed or inadequate assessment of the candidate by the examiners.

Introduction

When the result of the evaluation of a group of postgraduate medical residents is bad, it is either because the candidates are bad, or the questions and/or the mode of the examination are bad. It may also be the result of poor teaching and preparation of the candidates or the outcome of flawed assessment module. Unfortunately, in this type of situation, the blame is heaped on the candidates without considering the other factors mentioned above that may be responsible for such a poor result. In this write-up, the searchlight is beamed on the evaluators and the mode of evaluating1.

The purpose of this paper is to address the evaluators’ flaws in conducting postgraduate medical examination.

Reports

Material and Methods

Information used for this article was sourced from some relevant literature of English-speaking countries, the internet, by interviewing many experienced medical examiners and candidates who had sat the postgraduate medical examinations in various subspecialties, as well as the personal experiences of the authors in conducting such examinations.

Comments

Even though there may be some variations peculiar to an individual in evaluating a candidate for medical examination the objective of all evaluators is the same. The aims include, to have a good assessment of the candidate’s knowledge in the particular specialty following the laid down guidelines of the examining body.

Aggressiveness of the examiner.

One thing many examiners fail to realize is that even though the candidate for oral or clinical examination may appear calm and composed, the candidate is usually scared, uneasy and nervous2. All these are exaggerated when the examiner wears a scowl and looks aggressive. This demeanor impacts negatively on the performance of the candidate. In some cases the candidate may feel he/she is about to face the firing squad, and in some cases he may be tongue-tied or have a mental block just because of fear. Therefore the examiner should strive to present a friendly disposition to re-assure the candidate that he/she is not there “to get him”. After all, what the evaluator wants to know is how much the candidate knows and he is not his executioner. Therefore on first meeting the candidate, after confirming the candidate’s number, the next thing for the lead examiner to do is to introduce all the examiners to the candidate, and if possible engage the candidate briefly in an unrelated mundane discussion. This fleeting diversion tends to have a positive effect on the candidates.

Unnecessary interruption.

Unnecessary interruption of the candidate during a presentation not only interferes with the flow of thoughts and the structured discussion of the candidate it also disorganizes and distracts the candidate from what he/she is saying3,4. Even, if the candidate is saying what is considered as “nonsense” the examiner should be patient enough to hear all the “nonsense” the candidate is saying without any negative facial expression. After all, he reserves the right to score the candidate on his performance following the laid down marking scheme for the examination. However, occasionally the examiner may want to warn the candidate that he is on the wrong track. He can do this, in the case of a patient with a lump, by gently saying, “Do you really mean that there is no swelling present?” If the candidate does not yield to the subtle clue, the examiners need not harass the candidate any longer about the lump. However, occasional word or two of encouragement or reassurance if the candidate is getting it right is advisable. Although it may be difficult, an examiner should resist the temptation of converting the examination to a “teaching session”. For one thing, if the candidate did not know the information before the examination, it is highly unlikely that the “lesson” will be of much benefit to him at the examination session. Also, the “lecture session” might irritate the other co-examiners who had been waiting patiently for their turn to question and assess the candidate2

Disapproving facial expression and gesticulation

Even without talking, a negative facial expression, negative gesticulation, negative body language, hissing etc from the examiner convey a negative vibration to the candidate and this tends to destabilize the candidate. If the candidate is destabilized, the examiner cannot get a true picture of his knowledge which is what the examination is all about. This seemingly poor performance is examiner induced. It is therefore better for the examiner to maintain an expressionless face to whatever he considers “nonsense” the candidate may be saying. Even though some evaluators frown at a candidate presenting his case from a piece of paper, we see nothing wrong in this as long as the note was prepared by the candidate himself during the examination session. After all if the candidate needs the notes he made during the examination to remind himself of some facts, we see nothing wrong with this.

Summarising the clinical presentation.

Some candidates are in the habit of “summarizing” the case they have just finished presenting, and some evaluators are in the habit of asking for the “summary”. We have always wondered how a person can meaningfully “summarize” “History of presenting complaints; review of systems; past medical history; family and social history; obstetrics and gynaecological history; medications and allergies” which had just been fully presented a few minutes previously. Unless “summarizing” means, that the candidate should delete some of his findings. Yet everything is important in arriving at a diagnosis and the differential diagnoses, whether relevant negative findings or relevant positive findings. In the alternative, we recommend that the candidates be asked to highlight the important facts that point to the diagnosis. This will save time and lead to focused follow-up discussions and assessment.

Mixing Long case with Short cases.

A long case examination is meant to address at least three objectives - to establish the diagnosis,5,6 to detect any other pathology that may or may not be relevant to the obvious presentation by the patient and to show knowledge and skills on how to manage the case and its complications. Therefore, if the patient for a long case presented with a goitre or a breast lump, it behoves the candidate to fully examine the patient from head to toe. The reason for this is that if the patient complained only of thyroid swelling it is essential to be sure that he or she does not have any other condition like carcinoma of rectum or melanoma on the sole of the foot, which are conditions that carry worse prognosis than what the patient presents with. Not being a doctor, the patient may not know the significance of these additional pathologies or other features pointing to the extent of the disease, but only focuses on the neck swelling she is aware of. Therefore it will be wrong to focus the candidate’s attention only on the thyroid or the breast lump which is what the patient is complaining of when these are used for long case examination. If it is only thyroid or breast lump examination that is required, then the patients should have been used as short cases and not as a long case. Actually, the short case is supposed to represent a brief outpatient presentation and evaluation of a disease, while the long case replicates admitted patient being prepared for a surgical procedure.

Mixing the clinicals with the orals

There are essentially five parts to a post-graduate medical examination in West Africa. They are (i) the theory papers, (ii) the short clinical cases, (iii) the long clinical case, (iv) oral examinations and (v) the thesis/dissertation defence. This design is deliberate because it is an attempt to test every aspect of medical knowledge7,8,9 - thus each candidate is assessed in the cognitive, psychomotor skills and affective disposition for best practices.

The short case replicates the first, brief outpatient visit by the patient. The long case replicates an admitted patient being prepared for a surgical intervention or surgical management in the hospital. The orals assess principles of surgery, surgical pathology and operative surgery. Thesis/dissertation defence assesses the ability of the candidate to conceive and execute to successful conclusion a research problem. Therefore it is not very appropriate to, for example, during a short case examination of a goitre or a long case examination of a breast cancer for the examiners to ask the candidate how to perform thyroidectomy or a mastectomy. Such questions are meant for one of the oral examination sessions which deals with surgical pathology and operative surgery, and not for short cases or a long case examination. Strictly speaking, questions on the short cases should be limited to the examination, the diagnosis, the differential diagnoses and the general sketch of management of the pathology. Questions on the long case should also be limited to the clinical aspects of the case such as a full examination of the patient, differential diagnoses and relevant investigations for diagnosis, extent of the disease and the structured steps in the management of the pathology. To ask for operative details in a long or short case is to duplicate the questions meant for oral examinations. And this indicates repetition of questions which does not allow for a fair assessment of the candidate. Examiners, therefore, should keep strictly to the boundaries or limits of each section of the postgraduate examination.

Recommendations

To qualify to serve as a postgraduate medical examiner, the expert should attend prescribed train-the-trainers workshops on a regular basis and be assessed during the workshops. Only those who score a predetermined benchmark should be invited to serve as examiners. Also there should be adequate monitoring mechanisms in place to ensure compliance by the examiners with a view to delist examiners who turn a respected professional assessment guided by ethics into a theatre of the absurd for their self aggrandizement.

Conclusions

There is an erroneous belief that anyone who teaches can also conduct a proper medical fellowship examination. This is not always correct. In conducting a postgraduate medical specialty, the evaluators should have some appropriate and relevant training in medical assessment.10 Here are some guidelines to follow for a proper assessment of a candidate.

• 1 An evaluator should avoid a negative body language to the presentations and responses of the candidate under examination, irrespective of how inappropriate the answers may be. An expressionless demeanor, except for appropriate encouragement should be the guide.
• 2 There should be little or no interruption to what the candidate is saying. The occasional interruption should be to encourage the candidate if he is on the right track or to subtly hint him if he is on the wrong track. The evaluation should not be made another “lecture session” by the examiner. Also derisive statements and inappropriate language directed at the candidate should be avoided. The candidate, after the prescribed period of training and rotations, has come to be assessed so he can move on if he meets the requirement, but surely not to be derided. The treatment he is given is likely the type he will give to the residents who will train under him when he obtains the fellowship.
• 3There should be adherence as much as possible to what each section of the various parts of the examination is supposed to test.

Adherence to all these would give a fair assessment of the candidate’s knowledge in the index specialty.