The three Rs of written assessment: The JIPMER experience

Written assessment has stood the test of time and is widely practiced from elementary school to higher education. Medicine is not an exception and written exams are the main mode of assessing knowledge (cognitive domain) in formative and summative assessments. School education in India is much oriented towards the reproduction of knowledge rather than translation of knowledge. Hence, bright, curious, children are transformed to dull, indifferent parrots in school who get selected for various professional courses based on their ability to remember and reproduce facts. This concept of assessment is well conserved across different levels of education including medical education. The present assessment pattern in medical colleges, both undergraduate and postgraduate, maintains this status quo as we continue to expect students to cram stuff and reproduce it verbatim. This has also changed the way students learn the subject. Most of the students have now switched to third rated notes available in the market with only a very few reading standard textbooks. If we have to produce students who can analyze, think effectively, and treat patients appropriately, we need to change the way our students learn and the content of what they learn. The only effective way to do this is to change the way we assess them.

HOW TO CHANGE?

The present assessment is based on information transmission model rather than testing the real needs of a learner. Hence, the assessment should be designed to focus on learning outcomes with an intention to recognize deep approach to learning (understanding through an active constructivist engagement with knowledge) rather than surface approach to learning (reproducing through a passive incremental view of knowledge).[1] Following the three Rs to modify written assessment can bring a paradigm shift in the student's approach towards learning.

Reduce

The present day written assessment comprises mainly questions of recall. A practitioner needs to remember certain facts and figures by heart and we cannot do away with that. However, there are many other intellectual skills essential for a practitioner such as the analysis of information, synthesis of findings, application of concepts in a clinical context, evaluating the pros and cons of different modalities of treatment and selecting the most appropriate for a particular patient. Recall is at the lowest level of assessment of knowledge, and if we want to assess other intellectual skills, we need to ascend up in the ladder of assessment. Hence, the number of questions based on simple recall need to be reduced and instruments for assessing higher-order cognition (understanding, analysis, synthesis, application, and evaluation) should be introduced. In everyday practice, a doctor will be engaged in several iterations of analysis, synthesis, application, and evaluation. To be valid, the written assessment should be designed to incorporate all these evaluable intellectual skills.

Many times, there is considerable overlap in the content between two papers of a subject and also between different questions within a paper. This needs to be reduced. Considerable skewing of content is often noticed in written assessment. Questions are set according to the whims and fancies of the examiner without due consideration of the importance and weightage of every topic and this results in skewing. It is not fair on our part as examiners to design a skewed question paper. Every university should prepare a blueprint and follow it during the assessment. This will take care of content overlap and skewing of content to a major extent.

Refine

Assessment should be designed with the purpose in mind (purpose-driven assessment).[2] We need to continuously focus on the competencies expected of our students. Unfortunately, many faculty members harbor the notion that they should train the students to somehow get a pass (or a distinction!) rather than making them acquire skills (intellectual, behavioral, and psychomotor) required for the real world. The instruments (long/essay questions, short answer questions, etc.) used currently in the written assessment do not serve this purpose. Hence, we need to refine the purpose of designing every question in the formative and summative assessments. Before designing an assessment instrument, it is vital to define clearly what exactly is to be assessed. As far as possible, the long answer/short answer questions should be designed to mimic the real world scenario that would be encountered by the students when they become doctors. By doing so, our assessment tools will become more valid, meaningful, and will start serving the true purpose of driving learning in the right path. Moreover, students will start appreciating the relevance of questions asked in the assessment.

Students are asked to write short notes on a very broad topic. For example, “write short notes on antiplatelet drugs” or “write briefly on monoclonal antibodies” is a short answer question (6 marks) in a university examination. These questions are asked in formative/summative assessment for two reasons. The examiners are not competent enough to specify what really needs to be tested in this topic and it is very easy to set broad-based questions like this. Without making any intellectual effort, they can very easily set questions (and get paid for it). Second, most of the present-day examiners do not want any student to fail. While this is a laudable aspiration, the reason for it is not that laudable. If internal examiners fail a candidate, he/she will be coming back to them as intermediate batch and the faculty members have to take extra classes and other remedial measures for them. To avoid this, questions are so set that it becomes difficult for a student to fail. If a very broad topic is given as a short-answer question, whatever the student writes about the topic will fetch marks for them. It is time we realize that such questions totally defeat the purpose of assessment and there is no place for such questions in any form of assessment. Hence, we need to refine the focus of every question.

Replace

Pharmacology and therapeutics are like two sides of a coin. For a coin to be valid, both sides are required. Similarly, knowledge on pharmacology as well as therapeutics is essential for all practitioners. However, therapeutics is rarely taught (and hence assessed) in the pharmacology course for medical students. Students know the basic pharmacological points about a drug but never the therapeutic aspects of it. For example, students know the basic pharmacological points (mechanism of action, pharmacological actions, etc.) about digoxin, but they never realize that digoxin is the last drug to be started in a patient with congestive cardiac failure. Paradoxically, as most of the pharmacology textbooks (and our lectures too!!) start the chapter on drugs for heart failure with digoxin, students go with an opinion that digoxin is the first drug to be started in heart failure. Such things can be dangerous if left unaddressed. Similarly, students memorize the adverse effects of drugs as they are asked in the exams. However, they never bother to know what a physician should do, if a patient presents with an adverse effect to them during review – should the drug be stopped or the dose reduced or if the adverse effect is self-limiting and does not warrant any intervention, what should be done? How should the patient be reassured? Students do not learn these clinically relevant details as these are never assessed in the written exams. We need to start teaching therapeutics as a part of pharmacology and at least 40% of pharmacology should be replaced with therapeutics so that a balanced ratio of 60:40 between pharmacology and therapeutics is maintained while teaching and more importantly during assessment also.

This “replace” principle is not restricted to pharmacology alone. It is applicable to all pre- and para-clinical subjects. For example, in anatomy, morbid anatomy can give place to more of living anatomy. The reason being as doctors, we will be dealing with live persons and it becomes meaningless if a student knows the origin, insertion, blood supply of muscle but does not know how to test it clinically in a patient. Hardly, any such question is asked during the assessment in anatomy. Then how (and why) will the student ever learn such relevant points? In every subject during the assessment, replace redundancy with clinically relevant content. This will go a long way in the learning by students.

THE FOURTH R

By this time, many of you would have realized that the three Rs of written assessment are similar to that of ethics of animal experimentation. Rehabilitation was added as the fourth R to the latter but it is “rubrics” for written assessment. Even if we meticulously follow the principles of three Rs while constructing an instrument for assessment, the process of assessment can fail if the answer scripts are not evaluated properly. Rubrics as the fourth R can solve the problem. The preparation of rubrics should go hand in hand with that of the question paper. Preparation of model answers can give a clue about the inappropriately broad question with respect to the allotted marks and help in eliminating them while setting the question paper. During evaluation of answer scripts, rubrics help in bringing uniformity in the marking scheme and reduce inter- and intra-examiner variability. Thus, objectivity increases for essay/long answer and short answers questions with rubrics. Uniformity in scoring is very essential, especially with the increasing number of students. In formative assessment, it can help in providing feedback to the students.

OUR EXPERIENCE WITH THE THREE RS MODEL

There was a common opinion among the faculty members of our department that the standard of students is decreasing gradually and students were showing less interest in learning pharmacology. This was based on their observation and examination of students in formative and summative assessments. Most of our faculty members are well trained in medical education and they understood that unless the assessment is changed and made stringent, the standards could not be raised. We envisaged the three Rs model of written assessment and started implementing it. Initially, there was apprehension on the impact of this change on the marks obtained and the students’ acceptance of the same. To our surprise, students welcomed this change. They started understanding the relevance of the questions. Annexures 1 and 2 provide a sample question paper of our formative assessment based on the old and the new three Rs model. It is now two years since we implemented the change and all our formative assessment is based on the three Rs model. To make the written assessment more objective, we do adhere to rubrics, the fourth R. Every year, we get around 150 undergraduate medical students for the pharmacology course. Evaluating their answer scripts in a uniform and objective way for every assessment was a great challenge. Two examiners (faculty/resident) are assigned for evaluating a question with equal sharing of answer scripts. The answer key along with split up of marks is prepared, discussed in a meeting of all faculty members, and necessary modifications are made as per the suggestions. We (faculty and residents) gather in a hall for evaluating the answer scripts based on the prepared key. The process of following the rubrics has indeed brought more objectivity and uniformity in the evaluation besides being much quicker. The answer key along with the split up of marks is projected to the students while the answer scripts are distributed to them. All examiners provide feedback about the positive aspects and the common errors made by the students while answering the questions.

CONCLUSION

The three Rs model emphasizes us to reduce the number of recall questions and content overlap, refine the purpose of designing as well as the focus, and replace redundancy with clinical relevance. This model has been successfully implemented in pharmacology and would be applicable to all subjects. As the pattern of formative assessment lies with the department, without much difficulty, every department can implement the change at least in formative assessment.

ANNEXURE 1

ANNEXURE 2