Implementation of Objective Structured Practical Examination in Formative Assessment for Undergraduate Practical Pharmacology

Abstract

Context: At present, the method of practical assessment in pharmacology focuses on cognitive and subjective domains. Ideally, it should be objective and comprehensive.

Objective: To compare marks obtained in two types of exams, objective structured practical examination (OSPE) and traditional practical examination (TDPE), in order to assess faculty members’ and students’ perception of OSPE

Methodology: Sixty students having 75% attendance were included in the present study. Both OSPE and TDPE were conducted and marks obtained in each of the two exams were compared. Perceptions of students and faculty members were assessed on the basis of Likert- scale.

Result and discussion: The percentage of mean marks obtained in TDPE (67.5±2.24) was higher than in OSPE (66.5±2.78), but the results were not statistically significant (p=0.6). This may be due to first time exposure of OSPE and belief that marks would not be counted in internal assessment. Ninety five percent of students agreed that practical skills and knowledge acquired during OSPE would be also helpful after graduation, and 83.33% of participants admitted that personality, gender and other student-related factors did not affect scoring in OSPE. Eighty five percent of students and all faculties were willing to implement OSPE as an assessment tool. All faculties were in favour of OSPE due to its very low variability and because it motivated students learn skills as well subject more.

Conclusion:Objective structured practical examination is comparable to TDPE as assessment tool for pharmacology practical examination in terms of marks obtained by students, while satisfaction reported by faculties and students regarding various aspects of OSPE is much more in favour of the latter.

Keywords:assessment, objective structured practical examination, traditional practical examination, competency based under graduate curriculum.

INTRODUCTION

Medical education implies assessment of students at regular intervals as a source of learning and providing the basis for enhancing their competence level (1). Learning objectives cannot be reached without a uniform and reliable assessment method, and learner performance gives an indirect measure of teaching effectiveness. The present scenarios, mostly of Indian medical institutes, especially in pre- and paraclinical practical assessment method, focus on the cognitive and subjective domains, and they have a less predictive value due to a lack of proficiency in acquiring the required skills, frequently observed among our undergraduates. Students’ opinion reveal that teachers may have a subjective appraisal, marks may be due to luck, examination is less trusted. More error and mishaps in drug administration increase patient suffering and future doctors are less competent. Considering all these aspects, a national medical commission has been implemented in India (2).

It is well known that learning is assessment driven (3). No single assessment method is self-sufficient. There is an urgent need for implementation of an innovative method to assess medical graduates, as showed by a research paper entitled “Assessment for practical skills in medical education needs improvement from subjective methods to objective method” (4). In early 1975, an assessment tool to increase objectivity and conducting structured examination was developed in the University of Dundee (Dundee, Scotland) by Dr. Harden and his colleagues, which was suitably modified in 1979 (5, 6). Objective structured practical examination was the subject of an international conference organized in Ottawa in 1985, which provided the opportunity to share experience about OSPE and OSCE (7). In this conference, testing competencies in basic medical science, the term of OSPE is originated. All candidates are assessed using exactly the same stations and get marks for each step on the mark scheme that they perform correctly, which is therefore more prone to objective rather than subjective assessment. The candidate is given a very specific task, which is carefully structured to include parts from all elements of the curriculum as well as a wide range of skills. Students’ learning is influenced by the evaluation system. So, after exposure to OSPE, they will also focus on learning skills. Objective structured practical examination, a reliable and established practical examination system, is being used in many medical colleges in India and other developing countries (8-10), but it has been implemented for the first time in our institute.

The above-mentioned considerations motivated us to evaluate and plan an objective assessment method called objective structured practical examination (OSPE) in pharmacology practical exams. We chose the route of drug administration as main topic for the skill assessment practical test because drug administration is measurement issue causing substantial morbidity and mortality worldwide (11, 12). According to the study of Kumar KS et al., drug administration errors account for 15.34% of all medication errors (13). Intravenous (IV) drugs are most commonly associated with drug administration errors (11, 14, 15). Students would increasingly trust this assessment method, which would therefore become more reliable, objective and valid. Thus, with less errors in drug administration, more competent doctors will be produced.

Aims and objectives

• To evaluate score in term of marks obtained in OSPE compared to TDPE

• To evaluate perception of the second MBBS students and faculties towards OSPE

METHODOLOGY

Before starting this study, ethical approval was obtained from the institutional ethics committee. We invited all second MBBS of third semester students with more than 75% attendance in theory and practical classes. All eligible students were sensitized and clearly informed about the purpose of the present study. Subjects’ enrollment was done after obtaining their informed consent. Objective structured practical examination was conducted for the first time in our department, so detailed information of marks, number of stations, time for each station, etc, regarding this practical test was displayed on notice board prior to sensitization session for students. The study methodology is depicted in Figure 1.

OPSE assessment – The syllabus of assessment was selected from the third term in the field of pharmacology. It consisted of route for drug administration, dosage form, adverse drug effects and experimental pharmacology. Questions for OSPE station and questionnaire were prepared on the basis of previously published articles (1, 4, 8-10) and validated by faculties of our department. Based on their suggestion, the checklist and questionnaires were revised and finalized by consensus. Also, a pilot testing was done, in which two faculty members acted as assessors and one of the faculties served as observer to validate OSPE station.

After the two weeks of the sensitization session, students were subjected to OSPE with the help of faculties by validated questions and checklist. It was conducted through eight stations. There were six response stations, two procedure stations and one rest stations. Five minutes were given to each station, but marks allowed for each station varied according to each step of the checklist. Response stations were included to assess analysis, application and synthesis type of cognitive domain. Questions considered prescription writing, causality assessment of causative drug–adverse drug reaction pair, and dosage calculations. All exercises were provided in the form of a case based scenario to evaluate the level of cognitive domain. Procedure stations were included to assess psychomotor skills. Simulated patients and mannequins were used to assess drug administration skills. The rest station was included to help students get organised and relaxed. A total of 30 marks were allotted to OSPE.

Traditional practical examination (TDPE) – After two weeks of OSPE, all students appeared in TDPE at the end of the semester as part of the regular formative assessment. Traditional practical examination was conducted with similar syllabus and assessment contents without OPSE checklist. A total of 30 marks were allotted to TDPE.

Evaluation of assessment methods – After finalization of both parts, feedback from students and faculties was obtained to know their perceptions as well as to evaluate OSPE as a learning assessment tool using the five-point Likert scale (1=strongly disagree; 2=disagree; 3=neither agrees nor disagrees; 4=agree; 5=strongly agree.). For the ease of analysis, responses scored 1, 2 and 3 were merged as a ‘disagree’ category, and those scored 4 and 5 as an ‘agree’ category. Both categories were expressed as percentages. The total number of marks obtained in both assessment methods were expressed as mean ± standard deviation (SD) and were compared using paired t-test. Results and data were prepared on Microsoft Excel sheet, 2016.

RESULTS AND DISCUSSION

A total of 60 students participated in both exams. Their data was used to explore the results. There was a higher examination score in TDPE than OSPE. The percentage of mean marks obtained in TDPE (67.5±2.24) was greater than in OSPE (66.5±2.78), but the results were not statistically significant (p=0.6), as shown in Figure 2 and Table 1.

Of all participants, 83.33% agreed upon the fact that personality, gender and other student-related factors did not affect the score of OSPE exam; 95% of students believed that practical skills and knowledge acquired during OSPE would be helpful after graduation. More than 86% of students believed that questions were well sequenced and organized, whereas 80% experienced no stress during OSPE. Only 41.67% of students felt that it would have been exhausting to face a greater number of stations. Implementation of OSPE as an assessment tool in the future was welcomed by 80% of students. All findings were summarised in Table 2.

Perceptions of all participating faculties suggested that OSPE helped students acquire skills and knowledge in pharmacology, thus pledging for its implementation as an assessment tool in the future, the more so as it was also useful in diminishing examiner’s variability; also, OSPE was perceived to be more exhausting and lengthier if it had a greater number of stations (Table 3).

The main features of OSPE, that both the process and product are being tested, highlights the role of individual competencies (14). This method may not only improve the learning process but also prepare students to become good clinicians in the future.

In the present study, students experienced both TDPE and OSPE, and comparing the mean percentage marks revealed their perceptions of OSPE as an assessment tool by responding to questionnaires. We found a mean score 8% lower in OSPE than TDPE, but it was not significant (P=66), similarly to the results reported in a study conducted by Dr. Supriya et al. (4). A possible explanation might be that it was the firsttime exposure to OSPE and marks were not considered in internal assessment. Students had benefitted from prior exposure to OSPE, which had an acceptance rate of 73.33%.

Feedback suggested that OSPE was perceived by students as a reliable, effective, useful and well accepted method for assessment (16-18), which was relatively consistent with our findings (10, 16-18). Students in our study reported that OSPE was less stressful than TDPE, which was is contrast to many other studies. In our research, we saw an important difference between students and faculties regarding the number of OSPE stations; thus, 59.33% of students wanted more stations, while all faculties believed that an increased number of stations would lead to exhaustion. Except this divergence of opinion, feedback given by faculties was also in favour of OSPE.

Study limitations

• Students knew that marks obtained in OSPE were not included in internal assessment because they participated in a project study and also due to ethical issues; for this reason, students might have not taken things seriously.

• Conventional practical examination was conducted at two weeks after OSPE, which allowed more time for preparation.

• Students might have benefitted from OSPE due to same syllabus as both exams had a common topic.

CONCLUSIONS AND IMPLICATIONS

We concluded that OSPE was comparable to TDPE as a practical assessment tool in pharmacology in terms of marks obtained by students, while OSPE was superior to TDPE when considering the level of faculty and student satisfaction regarding various factors such as gender, personality difference and examiner variability, which were found not to affect the scoring of marks in examination.

Also, OSPE increases subjects’ motivation to acquire practical skills and knowledge; thus, it is accepted as an assessment tool in undergraduate practical pharmacology not only by students but also by faculties.

Conflict of interests: none declared

Financial support: none declared.

Acknowledgments: This research was conducted while the author was working as Professor and Head pharmacology at GEMRS Medical College, Gotri Vadodara. This paper was persented at the National conference on Health Professional Education 2016. We are very thankful to resource faculties of nodal Centre of MEU (NHLMMC) for their guidance to conduct and completion of this project as well as Dr. Jitendra Agrawal and our department staff for their help and support during this study.

FIGURE 1.

Study methodology

FIGURE 2.

Evaluation of marks in both types of examination

TABLE 1.

Examination scores in both types of exam

TABLE 2.

Students’ perceptions of OSPE (n=60)

TABLE 3.

Faculty members’ perceptions of OSPE (n=7)

ANNEXURE 1.

Some questions and checklist)

ANNEXURE 1.

Some questions and checklist)

ANNEXURE 1.

Some questions and checklist)

ANNEXURE 1.

Some questions and checklist)

ANNEXURE 1.

Some questions and checklist)