Evidence‐based medicine: assessment of knowledge of basic epidemiological and research methods among medical doctors

Abstract

Background

An understanding of statistical methods and basic epidemiology are crucial for the practice of modern medicine.

Aims

To assess (1) the knowledge of basic methods of conducting research and data analysis among residents and practicing doctors and (2) the effect of country of medical school graduation, professional status, medical article reading and writing experience on the level of this knowledge.

Methods

Data were collected by means of a supervised self‐administered questionnaire, which was distributed among doctors at Soroka Medical Center, Beer‐Sheva, Israel. The questionnaire included 10 multiple‐choice questions on basic epidemiology and statistics, and respondent demographical data.

Results

Of the 260 eligible doctors, 219 (84.2%) returned completed questionnaires. Of the 219 doctors, 50% graduated more than 8.5 years ago, 39.7% were specialists and the remaining were residents. The most frequent specialty was internal medicine (37.4%). Israel was the most frequent country of graduation (45.7%), followed by the former Soviet Union (Eastern medical education; 38.4%). The median total score of knowledge was 4 of 10 questions (interquartile range 2–6). A higher score was associated with a Western medical education, being a specialist, shorter elapsed time since graduation, higher number of publications and self‐reported reading of “methods” and “discussion” sections in scientific articles.

Conclusion

This study found a low level of knowledge of basic principles of research methods and data analysis among doctors, and this knowledge considerably differed by country of medical school graduation.

The main source of new knowledge for doctors in the era of evidence‐based medicine (EBM) is medical research results published in professional journals. According to EBM, diagnosis and treatment of patients should be based on data obtained from research studies that are rigorously designed and carried out. Nevertheless, there are numerous examples of medical studies with serious flaws in design, analysis and interpretation. It is possible to be seriously misled by taking the methodological competence of authors for granted.¹,² Surveys of medical literature estimate the rate of erroneous analysis and interpretation to be in the range of 30–90%.³,⁴,⁵,⁶,⁷ An analysis of reviewed papers submitted to one medical journal showed that among the most frequent and serious errors were determination of the study type, sample size considerations, selection of the study population, proper use of parametric tests, analysis of repeated measures, descriptive statistics and determination of confidence intervals (CIs). In addition, insufficient attention is paid to sample selection, participant refusal, data quality, choice of confounders and lack of p values.⁵,⁷ The median score (ranging from 0 to 5 points) measuring the quality of all articles reporting randomised clinical trials and published in four hand surgical and orthopaedic journals during 1992–2002 was only 2 points.⁸ A review of the medical literature published in 34 journals having the highest impact factor showed that 10% of the papers did not report on adjustment to confounders. The quality of the papers was found to be higher for researchers belonging to epidemiological, statistical and public health departments.⁹

To critically appraise published articles, doctors should have a basic understanding of the methods of epidemiology and biostatistics. These skills are particularly needed for conducting, analysing and reporting results of medical research.¹⁰ Several studies have found that doctors are often not fully competent in basic research methods. In a study testing 289 Canadian family practitioners for critical appraisal of methods and interpretation of results of medical articles, only 50% could satisfactorily answer the questions.¹¹ In another study, 50 Australian general practitioners were asked to explain the main concepts of EBM, such as relative risk and sensitivity.¹² It was found that even when doctors declared their knowledge of the subject, their answers were mostly wrong. Thus, one can conclude that a lack of knowledge in research methods is common among general practitioners. However, little is known about the understanding of basic statistical and epidemiology methods among doctors working in a university hospital who have an increased exposure to clinical research.

Medical education in Israel is research oriented; several courses on research methods are included in the curriculum of all four medical schools. By contrast, Israel is characterised by a high immigration rate: 16.3% of the population arrived in the country in past 15 years (nearly 90% from the former Soviet Republics).¹³ In 1990, the peak years of immigration, 3.9% of the immigrants were doctors and during the last decade the number of trained doctors in Israel more than doubled.¹⁴ Thus, a substantial number of practicing doctors in the country are foreign graduates who have predominantly arrived from the former soviet republic.

The main objective of this study was to assess the knowledge of the basic methods of research and data analysis among medical doctors in a university hospital. The secondary objective was to assess the effect of individual data (ie, professional status, country of medical education, and reading and publishing habits) on knowledge of research methods.

Materials and methods

Study population and data collection

Doctors working at the medical centre participated in the study. The medical centre has 40 departments and employs 500 doctors. Fifteen departments were chosen at random for recruiting study participants. The survey was distributed during the morning staff meetings of the departments. All doctors who agreed to participate in the study completed the questionnaire under the supervision of the primary investigator. The time allocated for filling out the questionnaire was limited to 15 min.

Assuming a difference in methods of scientific education across graduates from different countries, participants were divided into predefined groups according to the country of medical school graduation. The first group comprised doctors who graduated from the former republics of the Soviet Union (Eastern type of education). Doctors in the second group graduated from schools in all other countries including Israel, USA, Western (Germany, Italy, Netherlands) and Central (Romania, Czech Republic, Hungary) Europe and South America (Western type of education).

All participants were efficient in Hebrew (the questionnaire language)—an obligatory condition to enter a residency programme. The questionnaire was completed anonymously.

Questionnaire

The questionnaire (table 1) included 17 questions regarding (1) personal details of the doctors, the country and year of graduating from medical school, professional status (specialists v residents), and reading and writing habits, and (2)10 multichoice questions on basic research methods and statistics, five on each topic. The questions on statistics were based on the list of the most often used statistical methods published by Emerson and Colditz,¹⁵ who used this list in 1983 for a quality evaluation of the articles published in the New England Journal of Medicine and a similar article by Horton and Switzee.¹⁶ Missing answers to questions on epidemiological and statistical methods were considered to be incorrect.

Table 1 Questions testing knowledge of basic principles.

Questions testing knowledge of basic principles in epidemiology
Questions (short name)	Answers (correct answer underlined)
1. A study investigating an effect of a new drug for decreasing blood pressure should be a study of type: (RCT)	(a) Retrospective cohort study (b) Prospective case–control study (c) Double‐blind placebo–controlled study (d) Cross‐sectional observational study
2. You are investigating risk factors for a very rare disease. Which type of study you should choose in order to obtain results most effectively and quickly? (case‐control)	(a). Prospective cohort study (b) Case–control study (c) Clinical trial (d) Cross‐sectional observational study
3. Researchers compared between two diets by loss of weight measured at 3 months after the start of treatment. Study groups included 18 and 10 subjects. A decrease of 5% weight on average was observed in group taking diet A, and a decrease of 7% on average in group taking diet B. The observed difference was not statistically significant (p‐value>0.10). What could be the main reason for rejecting publication of these results? (power)	(a) Study groups are different by size (b) Results are not significant (c) The absolute difference in decrease in weight is very small (d) The power of the analysis is probably very small
4. You perform intention to treat analysis in summarizing the data of a clinical trial in order to avoid: (intention to treat)	(a) Recall bias (b) Selection bias (c) Verification bias (d) Lead‐time bias
5. Australian researchers found that excessive use of sun‐protective cream is related to development of skin cancer. This relationship could be partially explained by the presence of a confounder. To assess the direct effect of cream on development of skin cancer, the researchers should perform: (multivariate analysis)	(a) Adjustment to sun exposure by means of a multivariable analysis (b) Adjustment to sun exposure by excluding variable “sun exposure” from the multi‐variable analysis (c) New study in populations less exposed to sun (d) It is impossible to perform assessment of the direct effect of cream on development of skin cancer
Questions testing knowledge of basic principles in statistics used in medical studies
6. Treatment A was found to have a significant effect with p‐value = 0.05 and the treatment B effect was found significant with p value = 0.002. We may conclude that: (p value)	(a) The effect of treatment A is larger than that of treatment B
	(b) The effect of treatment B is larger than that of treatment A
	(c) It is impossible to compare the size of the effects
	(d) Both treatments have significant effect and therefore are equally effective
7. A researcher found an effect with p‐value = 0.07. A confidence interval of 95% (95% CI) for relative risk (RR) could be: (CI)	(a) 1.4 to 1.8
	(b) 0.3 to 0.9
	(c) 0.9 to 1.3
	(d) 1.2 to 2.5
8. Which test should be used for comparison of prevalence of disease A in men and women? (χ2 test)	(a) t test
	(b) χ2 test
	(c) Correlation
	(d) ANOVA
9. Which test should be used for comparison of blood pressure values between subjects belonging to three levels of smoking? (ANOVA)	(a) ttest
	(b) Paired t test
	(c) Correlation
	(d) ANOVA
10. A researcher compares satisfaction levels from treatment received in emergency department (measured in ascending categories from 1 to 4) between two study groups. Which test should be used? (non‐parametric test)
	(a) t test
	(b) χ2 tests
	(c) Correlation
	(d) Non‐parametric test

ANOVA, analysis of variance

To ensure validity of the questionnaire, the 10 questions assessing knowledge were given to 15 members of the Epidemiology Department, Ben‐Gurion University. All of them correctly answered all the questions.

Statistical methods

Differences between categorical variables were examined by the χ² test and Fisher's exact test, when necessary. The Mann–Whitney U test was applied to compare ordinal variables with multiple values, such as scores used in the study. For paired comparison of the scores, we used Wilcoxon's signed rank test. Correlation between continuous variables was estimated with Spearman's test. The multivariable analysis explaining the final score was carried out by a linear regression, when the estimates were adjusted to all variables found to be related in the univariable analysis with p<0.1. A p value = 0.05 was considered significant. Normally distributed variables are expressed as mean (standard deviation (SD)), non‐normally distributed variables as median and interquartile range (IQR).

Sample size considerations

The sample size calculations were based on the proportion of participants answering correctly ⩾5 of 10 questions. Assuming this proportion to be 50%, 95% CI and aiming maximal deviation from the estimate (7%), the sample size required was 196. Owing to the expectation of a low compliance rate, the final sample size was deliberately increased.

Results

During the study period, 260 doctors were recruited and 226 (87%) returned completed questionnaires. Of 226 returned questionnaires, 7 contained missing data in the personal details questions. Thus, the final analysis included 219 doctors, of whom all personal details were available.

Table 2 depicts doctors' professional characteristics according to the country of medical school graduation. Of 219 participants, 84 (38.4%) graduated from the former Soviet republics. The remaining 135 doctors were distributed by the country of graduation as follows: Israel, 100 (45.7%); West and Central Europe, 22 (10.0%); Italy, 8; Germany, 3; Czech Republic, 3; Hungary, 3; Netherlands, 1; Romania, 4; South America, 10 (4.6%); Argentina, 5; Cuba, 3; Uruguay, 1; Brazil, 1; and North America, 3 (1.4%). The time elapsed from graduation was shorter among doctors from these countries compared with doctors who graduated in the former Soviet Union—8 years (IQR 4–19) versus 10 years (IQR 6–19), p = 0.02. There were less specialists among foreign graduates compared with doctors who graduated from Israel—32.8% versus 48.0%, p = 0.02. There were differences in choice of residency between the two groups: 29.3% of the domestic graduates choose paediatrics or obstetric and gynaecology compared with only 9.3% among foreign graduates, p<0.001.

Table 2 Medical education and professional characteristics stratified by country of graduation.

	Total, n = 219	Eastern medical education, n = 84	Western medical education, n = 135	p Value
Years after graduation
Median	8.5	10.0	8.0	0.57*
Interquartile range	(5–19)	(6–17)	(5–20)
Professional stage
Residents, %	60.3	73.8	51.8	0.004
Specialists, %	39.7	26.2	48.1
Primary specialisation†, %
Internal medicine	37.4	36.7	37.6
Family medicine	16.2	20.3	13.6
Surgery	12.3	13.9	11.2
Paediatrics	9.8	6.3	12.0
Obstetrics and gynaecology	8.8	2.5	12.8
Anaesthesiology	5.4	8.9	3.2
Roentgenology	4.9	7.6	3.2
Neurology	5.4	3.8	6.4

*Mann–Whitney U test.

†Internal medicine, three departments; surgery, two departments; paediatrics, three departments; obstetrics and gynaecology, three departments.

Table 3 describes the reading and publishing habits of the participants. A total of 96% of the participants reported reading at least one article per week, whereas 35.2% usually read at least three articles. Specialists read significantly more articles per week—52.3% of them read at least three articles, compared with only 23.8% of the residents; p<0.001. Most of the doctors, 63.6%, participated in the writing of ⩽5 articles. Similar to the reading pattern, only 21.1% of the residents wrote ⩾6 articles compared with 44.0% of the specialists; p<0.001. The Spearman correlation value between reading and writing variables was 0.35; p<0.001.

Table 3 Habits of reading and publishing of medical articles.

	Total, n = 219	Eastern medical education, n = 84	Western medical education, n = 135	p Value
Number of papers read/week, %
0	3.7	3.6	5.8	0.15
1–2	61.1	68.7	56.4
3–5	23.6	22.9	24.1
6–10	7.4	2.4	10.5
>10	4.2	2.4	5.3
Number of published papers, %
0	26.6	38.6	20.7	0.003
1–5	37.0	40.4	35.3
6–10	12.7	8.8	14.7
11–15	1.2	3.5	—
>15	22.5	8.8	29.3
Parts of a paper usually read, %
Abstract	86.9	74.4	94.5	<0.001
Background	55.8	46.2	61.7	0.03
Methods	40.5	30.8	46.5	0.03
Results	70.9	59.0	78.1	0.003
Discussion	80.6	76.9	82.8	0.30

*Mann–Whitney U test.

According to the reports of the participants, the abstract was the most frequently read section of the article (86.9%), and the methods was the most rarely read (40.5%). In all, 14 (6.4%) participants reported reading the only abstract. Those who read the methods section were more likely to read ⩾3 articles per week, compared with those who did not (53.6% v 21.2%; p<0.001). The doctors who read ⩾3 articles per week had a higher rate of writing ⩾5 articles during their career (53.7% among those who read the methods compared with 23.3% among non‐readers; p<0.001).

Figure 1 illustrates the results of the participants in the 10‐question test on epidemiological and statistical methods. The rate of the correct answers to the individual questions ranged from 84.9% to the question on randomised control trials to 24.7% on the question requiring knowledge of analysis of variance methods. The median total score (total number of correct answers) was 4 of 10 (IQR 2–6). The score on the five questions on epidemiological methods (median 3, IQR 2–4) was significantly higher than that on the five questions on statistical methods (median 2, IQR 1–3); p<0.001 (paired Wilcoxon signed rank test). The total score of the graduates from non‐Soviet schools (median 3, IQR 2–4) was higher than those from the former Soviet republics (median 2, IQR 1–3); p<0.001. Those who read the methods part of the article had higher total scores, compared with those who did not—median 5, IQR 4–7 v median 4, IQR 3–5; p<0.001. Similarly, reading the discussion was associated with higher scores—median 5, IQR 3–6 v median 4, IQR 2–5; p = 0.03.

Figure 1 The figure describes the average of correct answers to 10 questions in understanding different aspects of basic research methods. Two populations of doctors are compared: those who graduated in the former Soviet Union (Eastern type of education) and those who graduated in Israel, USA, Western and Central Europe, and South America (Western type of education). RCT, randomised controlled trial; CI, confidence interval; ANOVA, analysis of variance.

The proportion of doctors correctly answering ⩾5 questions was 45.6% (95% CI 39.0% to 52.3%). Table 4 describes characteristics of the participants stratified into two groups: those with total score ⩾5 and those with a total score ⩽5 correct answers. Higher total score was associated with higher numbers of articles read per week (Spearman correlation 0.25; p<0.001), and the number of published articles (Spearman correlation 0.41; p<0.001). No significant correlation was found between the number of years elapsed from graduation and the total score in univariate analysis (Spearman correlation 0.06; p = 0.400).

Table 4 Characteristics of the study population according to the number of correct answers.

	⩾5 Correct answers, n = 103	<5 Correct answers, n = 116	p Value
Years after graduation			0.81*
Median	9	8
IQR	4–20	6–16
Professional stage			<0.001
Residents, %	47.6	71.6
Country of graduation			<0.001
Western medical education, %	60.7	39.3
Eastern medical education, %	25.0	75.0
Number of papers read/week, %			0.01
0	3.0	4.3
1–2	52.5	68.7
3–5	25.7	21.7
6–10	10.9	4.3
>10	7.9	0.9
Number of published papers, %			0.001
0	16.1	37.2
1–5	34.5	39.5
6–10	13.8	11.6
11–15	2.3	—
>15	33.3	11.6

*Mann–Whitney U test.

Table 5 shows the results of multivariate analysis for prediction of the total score. Number of years after graduation and medical education in the former Soviet Union (Eastern type of education) reduced the total score after adjusting for professional status, reading and writing habits. The linear model was further applied with the following breakdown of the country of graduation into three groups: Israel and Western countries (USA, Western Europe), former Soviet Union and all other countries (Central Europe and South America), with Israeli graduates serving as a reference group. The analysis showed that although former Soviet Union graduates performed poorly (standardised coefficients β  = −0.19; p = 0.012), there was no difference between the other two groups (standardised coefficients β for Central Europe and South America  = −0.3; p = 0.720).

Table 5 Linear regression for prediction of the total number of correct answers.

	Standardised coefficients β	p Value
Years after graduation, per year	−0.35	<0.001
Number of publications	0.33	<0.001
Specialists v residents	0.31	0.002
Eastern (former Soviet Union) v Western medical education	−0.18	0.01
Reading of methods part	0.17	0.02
Reading of discussion part	0.14	0.04

Discussion

This study can be summarised by two main findings: (1) a low level of knowledge in the basic principles of study methods and data analysis among doctors and (2) a significant variance in knowledge by country of medical school graduation.

The first finding was not totally unexpected. Preceding studies conducted among family practitioners in Australia and Canada showed a serious gap in knowledge of medical research methods.¹¹,¹² The previously published data were obtained from assessing the knowledge level in a population of general practitioners. We could theoretically expect a higher level of knowledge among doctors working in a university‐affiliated hospital, where research activities are more intensive. Nevertheless, this study showed that half of the participants correctly answered ⩽4 questions.

We found that a better knowledge was increasingly associated with (1) more recent graduation; (2) specialist training; (3) participating in more publications; (4) reporting of reading Methods and Discussion sections; and (5) graduation from a Western‐oriented medical school compared with one from the former Soviet Union. At first glance, the first two predictors seem to contradict each other, although they come from the same multivariable model, and therefore, are adjusted to one another. This finding probably suggests that modern medical education provides more training in epidemiology and research methods than older curriculums. Notwithstanding, a limited body of evidence suggests that teaching critical appraisal skills has a substantial effect on knowledge of research methods.¹⁷,¹⁸,¹⁹ A Cochrane review looking at the effectiveness of teaching critical appraisal skills to health professionals found only one study that met their inclusion criteria, which showed that critical appraisal teaching had a positive effect on the participants' knowledge. In that study, critical appraisal teaching resulted in a 25% improvement in critical appraisal knowledge compared with a 6% improvement in the control group.²⁰ We found that prolonged exposure to research, which characterises specialists, increases the level of knowledge of research methods, compared with residents who are at the beginning of their careers. The third predictor, defined as the number of publications, is consistent with the status of a specialist, who more often participates in medical studies and publishes research findings. Thus, it may be concluded that a prolonged exposure to research activity, rather than the formal learning of the discipline, is more important for doctors in attaining the appropriate level of knowledge.

A higher level of knowledge in research methods was associated with self‐reporting reading of Methods and Discussion sections of papers. This fact can be explained in two opposite ways: (1) that reading of these sections improves epidemiological knowledge and (2) that a prior knowledge of research methods makes the Methods and Discussion sections more easily understood and, therefore, more attractive for a reader. Being so ambiguous, this factor does not substantially add to our understanding of knowledge of research methods among doctors.

We found that the country of medical school graduation is significantly related to knowledge of the methods. This difference becomes even more striking when the countries are grouped by their methods of teaching medicine: Western and Eastern. This phenomenon deserves special attention, as it may be relevant to any country facing immigration and the absorption of foreign medical doctors in its health system. A recent survey showed that immigration of doctors from developing countries to the Western countries is a common phenomenon.²¹ On the basis of the findings of this study, medical doctors who graduated from medical schools in the former Soviet Union are expected to have less knowledge of research methods, thereby affecting their ability to practice in a modern healthcare environment with its emphasis on self‐directed learning. As publishing in international medical periodicals was infrequent before the former Soviet republics were opened to the West, doctors coming from this region usually have never been exposed to accepted research methods. This fact may influence their inclination to participate in, and use the results of, medical research, as well as their knowledge of the subject. Similar results describing limitations of research methods in medical journals published in China (an important source of migrating doctors) showed that outside Western countries inadequate medical research methods remain a serious problem.²²

Limitations

Ths study was limited to one medical centre, which is probably not fully representative of all medical centres in Israel. The data on manuscript reading and writing habits were collected by the self‐reporting of doctors and therefore are very subjective. However, the reading and writing habits are expected to be an overestimation of the unknown true figures. This limitation should not affect the second finding of the study regarding heterogeneity in research knowledge by the country of medical school graduation. The main limitation can be the language difficulties of non‐Hebrew speakers (foreign graduates) in answering the questionnaire, which can potentially confound the results of our study. On the other hand, all foreign graduates accomplished a special 1‐year linguistic course and a 6‐month medical course in Hebrew before the license examinaton. Furthermore, graduates from Central Europe and South America scored significantly higher than graduates from Eastern (former Soviet Union) countries and equalled the graduates from Western countries , although Hebrew was not the primary language for both populations. Furthermore, most of the medical literature is published in English. Although our questionnaire was given in Hebrew, all “special” terminology generally used in medical papers (power, intention‐to‐treat, etc) was also translated to English. Thus, a university‐based hospital doctor can be expected to be familiar enough with this terminology regardless of mother tongue.

Conclusions

We have shown that doctors working in a university hospital had a limited knowledge of the main principles of research and data analysis, and therefore are not capable of using them effectively in clinical practice. The absence of this knowledge can also negatively affect a doctor's ability to conduct research. This problem is especially prevalent among doctors who graduated from medical schools in the former Soviet Union. These doctors present a target population for additional introductory courses in the principles of research, as a part of their absorption in Western‐oriented medical centres.

Abbreviations

EBM - evidence‐based medicine

IQR - interquartile range