Experience of Teaching Critical Appraisal of Scientific Literature to Undergraduate and Postgraduate Students at the Ziauddin Medical University, Karachi, Pakistan

Abstract

Background:

Critical appraisal of scientific literature is an integral part of Evidence Based Medicine (EBM). Many medical practitioners have either limited or no formal education in research and are inadequately prepared to critically analyze the quality of research they are reading. This study presents the instructional strategy, students’ evaluation and the feedback of the undergraduate and postgraduate students on teaching critical appraisal of published medical literature to undergraduate and postgraduate students in the Ziauddin Medical University, Karachi, Pakistan.

Subjects:

Two batches of undergraduate medical students of Year-3 (n = 85) and a group of (n = 18) postgraduate students in basic sciences, community health sciences and family medicine.

Methods:

After 170 hours of teaching of biostatistics, epidemiology and survey methodology in Year-1 & 2, in Year-3 of undergraduate curriculum, six 2-hour structured sessions for critical appraisal of research articles published in peer reviewed journals were held.

Results:

All (N=103) students who took the course appeared in the objective structured practical examination (OSPE), where out of 100 they scored 74.3 ± 9.1. The studentds’ feedback on a 5-point Likert’s scale questionnaire showed the mean of overall satisfaction of the students is 3.93, and appreciation of relevance of quantitative subjects to understand medical literature is 4.89. All respondents agreed and strongly agreed the course helped them appreciate the relevance of quantitative subjects to understanding of medical literature

Discussion:

This course should be considered as the first step in the journey of becoming a competent self learner and should be followed by courses on EBM.

Introduction

Medical graduates need continually updating themselves to keep abreast of relevant exponentially expanding knowledge and research on biomedical and technological advancement.⁽¹⁾ Self directed learning (SDL) has become imperative. The most convincing statement regarding SDL that is often quoted is of Dr Sydney Burwell, Dean Harvard Medical School. He says “my students are dismayed when I say to them that half of what you are taught as medical students will in 10 years been shown to be wrong. And the trouble is none of your teachers know which half”.⁽²⁾ Physicians are continuously inundated with new information about preventive measures, diagnostic tests and treatment that will improve quality of patient care.⁽³⁾ Internet searches result in a very large number of articles on a single topic. Results, interpretation and recommendations of studies cannot be taken at face value since they may have weak research methodologies or inappropriately presented results and / or recommendations.⁽⁴⁾ Hence, physicians need to be able to critically evaluate the validity and soundness of the research study(ies) for considering application of therapeutic, preventive and rehabilitative procedures, in their settings. ⁽⁵⁾ Currently there is growing demand for Evidence Based Medicine (EBM) and it is being promoted for improving quality of patient care. ⁽⁶⁾ Critical appraisal of scientific literature is an integral part of EBM. A study from the West has revealed that more than 50% of medical schools graduates report inadequate training in critical appraisal of biomedical literature. ⁽⁷⁾ Many medical practitioners have either limited or no formal education in research and are inadequately prepared to critically analyze the quality of research they are reading. ⁽⁸⁾ A study from Canada also reported poor critical appraisal skills in family physicians.⁽⁹) Another prospective randomized controlled trial concluded that “one-off” educational interventions directed towards enhancing evidence based practice is not very effective and is also costly. ⁽¹⁰⁾

Numerous published studies in both general and specialist journals have shown weak or inappropriate study designs, misinterpretation of results, selective reporting of results, citing of literature selectively and drawing unjustified conclusions.^(11–17) Undergraduates and many medical graduates take findings and conclusions of published studies without questioning the soundness of the research, perhaps because of their ignorance in statistics and epidemiology or lack of critical appraisal skills. Responsible medical journals get statisticians to carefully scrutinize the research studies submitted for publication. ⁽¹⁸⁾ Unfortunately many do not and bad papers get published. ⁽⁴⁾ “Poor quality of much medical research is widely acknowledged, yet disturbingly the leaders of the medical profession seem minimally concerned” ⁽⁴⁾ Failure to appreciate the basic principles underlying scientific research and not enough emphasis on statistics and epidemiology in medical education coupled with the climate of “publish or perish” are some of the main issues that need to be addressed. ⁽⁴⁾

Many medical practitioners have either limited or no formal education in research and quantitative methods that would prepare them to critically analyze the quality of research they are reading. ⁽⁶⁾ The teaching of critical appraisal, a major component of EBM, has been increasingly integrated into all levels of medical school curricula. ⁽⁷⁾

This study presents the instructional strategy, students’ evaluation and the feedback of the undergraduate and postgraduate students on teaching critical appraisal of published medical literature to undergraduate and postgraduate students in the Ziauddin Medical University (ZMU), Karachi, Pakistan.

Methods

Site: The Ziauddin Medical University was established in 1996 as the first problem-based medical institution with integrated curricular approach in Pakistan.

Subjects: Two batches of undergraduate medical students of Year-3 (n = 85) and a group of (n = 18) postgraduate students in basic sciences, community health sciences and family medicine.

Methods: The Department of Community Health Sciences (CHS) conducts courses in Epidemiology and Biostatistics in Year 1 & 2 of the five-year undergraduate medical curriculum. Total teaching hours allocated to Biostatistics (Basics &Inferential), Epidemiology (Descriptive & Analytic) and Survey Methodology is 70 hours, 70 hours and 30 hours respectively. These courses are mandatory for the postgraduate also to pass and they attend the sessions along with the undergraduate students. In Year-3 of undergraduate curriculum, six 2-hour structured sessions for critical appraisal of research articles published in peer reviewed journals were held. After an introductory 1-hour lecture on checklist (Table 1), students (under-graduate and postgraduate) were divided in small group of 8 to 10, facilitated by a CHS faculty member. Articles for the session were chosen by the lead facilitator of the course and were given to the students one week in advance to allow sufficient time to read. The lead facilitator prepared a list of 20 to 25 questions pertaining to different sections of each selected published paper as a guide for the facilitators for the session. These questions were indirectly raised only when the students missed some critical points in the article. At the end of the course students’ were evaluated by OSPE in which there were 5 stations for different sections of the paper on which they were required to answer questions in an allotted time (12 minutes) for each station. The published research papers given in the examination were different than the ones discussed in the small group sessions. Feedback from the students was taken on a 5-point Likert’s scale on 12 items at the end of the course after the OSPE. Analysis was done by both frequency distribution and percentages of total responds of each item, as well as mean of each item.

Table (1). Check list for critical reading.

EXTERNAL VALIDITY:
Journal:	Is the journal indexed / peer reviewed?
Title:	Does the title indicate the material in the article ?
	Is it indicative of the subject you are interested in ?
Author/s:	Who are the author/s
Institution:	Which Institution(s) do the author/s belong to ?
Abstract:	Does the abstract interest you ?
	Does it give a good summary of the article?

INTERNAL VALIDITY:
Introduction:
Literature:	Is the literature review well done ?
	Is it relevant to the subject ?
Rationale:	Is the rationale of doing the study stated?
Hypothesis / Objective(s):	Is it well stated / formulated?
Materials and Methods:
Setting:	Is the setting well defined?
	Is the setting of relevance to the objectives?
	Is the setting of interest to you?
Design:	What is the study design?
	Is it appropriate to answer/address the research question/hypothesis?
Subject:	What is the sampling method?
	Is it well stated?
In/Exclusion criteria:	Are they well defined?
Selection bias:	Are the measures to remove selection bias stated or undertaken?
Other biase Biases / confounding factors:	Were any measure to remove or control for confounders undertaken?
Statistical methods:	Were appropriate methods applied?
	Are the p-value / confidence interval given?
Results:
Data presentation: Is it appropriate? Is there any missing data? Do the numbers add up? Are the main findings stated appropriately?
Discussion & Conclusions: Is it relevant and consistent with the literature review? Are limitations, if any, stated? Are the recommendations appropriate?

Results

All (N=103) students who took the course appeared in the OSPE, where out of 100 they scored 74.3 ± 9.1. Students were asked to give their feedback on the course by filling a questionnaire and rate each of the 12 items on a 5-point Likert’s scale using the following: “Strongly Disagree” = 1, “Disagree” = 2, “Somewhat Agree” = 3, “Agree” = 4, and “Strongly Agree” = 5. Using frequency analysis, the table 1 summarizes students rating and the corresponding percentages of total respondents for each of the 12 items. In addition, the mean rating for each item are also listed in the table 2.

Table (2). Students’ Feedback on Critical Reading Course.

N = 103

1. SD = Strongly Disagree, 2. D = Disagree, 3. SWA = Somewhat Agree, 4. A = Agree, 5. SA = Strongly Agree

	SD 1	D 2	SWA 3	A 4	SA 5
	n (%)	n(%)	n(%)	n(%)	n(%)	Mean
1) The course helped me to understand relevance of Epidemiology &biostatistics to comprehend research articles				11 (10.7)	92 (89.3)	4.89
2) Time allotted to the course was adequate	2 (1.9)	8 (7.8)	21 (20.4)	34 (33.0)	38 (36.9)	3.95
3) Faculty support to students was adequate			17 (!6.5)	23 (22.3)	63 (61.2)	4.44
4) Articles selected was appropriate for students			3 (2.9)	6 (5.8)	94 (91.3)	4.88
5) Timing of the course is right			12 (11.6)	21 (20.4)	70 (68.0)	4.56
6) After the course I am more comfortable in critically appraising the “Introduction section” of articles				6 (5.8)	97 (94.2)	4.88
7) After the course I am more comfortable in critically appraising the “Methodology section” of articles		2 (1.94)	13 (12.6)	34 (33.0)	54 (52.4)	4.36
8) After the course I am more comfortable in critically appraising the “Results section” of articles		14 (13.6)	22 (21.5)	36 (34.9)	31 (30.0)	3.81
9) After the course I am more comfortable in critically appraising the “Discussion & conclusions” section of articles			4 (3.9)	9 (8.7)	90 (87.4)	4.83
10) Students’ evaluation by OSPE was well designed			17 (16.5)	39 (37.9)	47 (45.6)	4.29
11) I found the course was useful for understanding research articles				5 (4.9)	98 (95.1)	4.95
12) Overall I am satisfied by the course		1 (0.9)	11 (10.7)	23 (22.3)	68 (66.0)	3.93

The mean of the overall satisfaction of the students is 3.93, and appreciation of relevance of quantitative subjects to understand medical literature is 4.89 (Table 2). All the respondents agreed and strongly agreed that a) the course helped them appreciate the relevance of quantitative subjects to understanding of medical literature; b) respondents felt more comfortable with appraising the “introduction” section of articles and c) they found the course useful for understanding research articles. Sixty five percent respondents felt comfortable in appraising the “result” section of articles, while 83.5% agreed and strongly agreed that the OSPE was well designed. Four-fifths (88.3%) of the respondents agreed and strongly agreed that the timing of the course was right (Table 2).

Discussion

Reading medical journals is the most important activity for keeping updated, ranking above textbooks, colleagues, continuing education courses, and pharmaceutical representatives. ⁽¹⁹⁾ The results of the students feedback on the course and their performance in the summative evaluation has been encouraging. However, one-third of the students were not still comfortable in critically appraising “Results” section; this needs to be taken seriously, because the articles selected were simple and the statistics used in these studies were previously taught to the students in the biostatistics course. Therefore, teaching-learning strategy for the preceding courses should be re-evaluated and further strengthened. The critical appraisal of medical literature course following biostatistics and epidemiology courses was highly appreciated by the students. Experience of critical appraisal of medical literature delivered at the Department of Community health Sciences at the Aga Khan University (AKU), Karachi, Pakistan reported that all students agreed that critical reading skills were essential, but only 30% strongly agreed that they had mastered the skills and 97% students agreed that year 3 was appropriate to start critical reading. ⁽¹⁾ Results and feedback of both under- and postgraduate students are given together is because both took this course for the first time.

In Pakistan, unfortunately only 2 medical schools, AKU and ZMU teach this course as mandatory part of their medical curricula. Teaching of Evidence Based Medicine courses is a logical progression from teaching critical appraisal of medical literature; however, due to lack of faculty capacity this initiative has yet to be undertaken at ZMU.

Conclusion

In view of the above cited unfortunate realities, students as well as medical practitioners should pay attention to acquiring the skills for critical appraisal of medical literature. It is the responsibility of the medical schools to impart basic skills in this much neglected, but very important area of medical education. At the same time medical practitioners should be trained not to accept published medical research studies at face value but to form their own conclusions after critical review. The effort is worthwhile in investing time for becoming better self learners and updating ones knowledge and skills. The above mentioned ZMU course should be considered as the first step in the journey of becoming a competent self learner.