Methodological Errors in Clinical Studies Published by Medical Journals of Ex-Yugoslav Countries

Abstract

Introduction:

Certain methodological principles should be inexcusably followed when designing clinical or observational research to avoid bias and presentation of results that do not reflect the truth about the phenomenon that is the object of the study.

Aim:

The aim of this study was to compare the methodological quality of clinical trials and observational studies published in medical journals from ex-Yugoslav countries indexed in Pubmed/MEDLINE.

Methods:

Clinical studies published in medical journals of ex-Yugoslav countries were retrieved from the Pubmed/MEDLINE database, and the sample for analysis was randomly chosen from the retrieved publications. The rate of the most common errors in the design of clinical/observational studies was established by a careful reading of the sampled publications and their checking against predefined criteria.

Results:

The studies published in two countries that are now member states of the European Union (Slovenia and Croatia) have significantly higher citation rates, impact factor, and methodological quality scores than studies from other ex-Yugoslav countries. While publications from Croatia show clear improvement trend throughout the last two decades, which is visible also in the last 10 years in Slovenia and Bosnia and Herzegovina, quality of clinical research published in journals from Serbia was stagnating in the same period.

Conclusions:

There are significant differences in methodological quality and scientometric characteristics of clinical research published in medical journals of ex-Yugoslav countries that could be mitigated by more intensive training of clinical researchers in statistics and research design, as well as by more rigorous editorial practices.

1. INTRODUCTION

“Most people say that is the intellect which makes a great scientist. They are wrong: it is character.”

(ALBERT EINSTEIN)

True knowledge is gained through scientific research. Scientific publishing is one of the most important and influential results of scientists work. The number of publications and their quoting are measures of scientist success while unpublished researches are invisible to the scientific community, and as such nonexistent. A scientific paper by its publication becomes a permanent heritage which is subject to review and critique (1). The highest degree of knowledge is the ability to investigate scientific problems. The best way to get to the truth is knowledge of scientific methods. Finding truth in science is not only ensured by the consistent application of scientific methods. The honesty of a scientist who searches for truth is of crucial importance. Also, teamwork today is almost a prerequisite for success in scientific work, so modern science is multidisciplinary and multi-professional, so that the work resulting from this cooperation is usually signed by several authors. Author(s) signed the scientific work and is fully responsible for all its contents. The question of authorship is of great importance because it is crucial in the career of a scientist. Scientific publication is not just a way of scientific communication, but also the basis of the evaluation of scientists. Public disclosure of the results of scientific research is the basic postulates of the scientific method and their basis on the doctrine of truth. Scientific and professional work is primarily an educational tool and its content can be presented in different ways. Scientific papers and articles are highly specialized manuscripts on research published in indexed scientific journals (1).

Certain methodological principles should be inexcusably followed when designing clinical or observational research to avoid bias and presentation of results that do not reflect the truth about the phenomenon that is the object of the study. The phenomenon under study should be measured by previously validated instruments, the study sample should be of sufficient size to avoid false-negative results, certain statistical tests should be used only if their assumptions were met, and caveats of the study should be clearly and frankly stated, so the readers can draw correct conclusions (1). However, although the quality criteria for clinical and observational studies are well known and undisputed, they are still not followed completely, as several recent studies have demonstrated (2-4).

Almost three decades passed from the dissolution of Yugoslavia, and the states that emerged from its republics followed different economic, cultural, and social development pace. The differences are also visible in healthcare and medical science (5-7), and reviewing them allows learning about appropriate and less appropriate governing decisions, and their consequences. The editorial practice of medical journals published in ex-Yugoslav countries and visible internationally is of prime importance for directing and shaping clinical research within the region, and insisting on sound methodological principles could improve its quality and generalizability (8). It is relatively easy to get an insight into scientometric differences among published clinical research studies from various countries, but analysis of their methodological quality is also necessary to understand reasons for these differences and foresee improvement pathways (9).

2. AIM

The aim of this study was to compare the methodological quality of clinical trials and observational studies published in medical journals from ex-Yugoslav countries indexed in Pubmed/MEDLINE.

3. METHODS

The studies were retrieved for analysis from the PubMed (MEDLINE) database. The following inclusion criteria defined for each Ex-Yugoslav country separately the pools from which the study samples were extracted: journal article, published in a journal issued in Slovenia, Croatia, Bosnia and Herzegovina, Serbia, Montenegro or North Macedonia, original clinical trial, original observational clinical study, and full-text availability. The exclusion criteria were: review articles, animal studies, in vitro studies, modeling studies and case reports or case series. To define search strategies, at first, the following journals published in each of Ex-Yugoslav countries and indexed in PubMed (MEDLINE) were identified in the National Library of Medicine’s Catalogue of journals indexed in MEDLINE (10, 11): Slovenia: Acta Dermatovenerologica Alpina, Pannonica, et Adriatica, Radiology and Oncology, and Zdravstveno Varstvo; Croatia: Acta Medico-Historica Adriatica, Acta Dermatovenerologica Croatica, Biochemia Medica, Acta Pharmaceutica (Zagreb, Croatia), Croatian Medical Journal, Acta Clinica Croatica, Psychiatria Danubina, and Arhiv za Higijenu Rada i Toksikologiju; Bosnia and Herzegovina: Acta Informatica Medica, Materia Socio-Medica, Medical Archives, Acta Medica Academica, Medicinski Glasnik, Bosnian Journal of Basic Medical Sciences; Serbia: Acta Chirurgica Iugoslavica, Medicinski Pregled, Vojnosanitetski Pregled, and Srpski Arhiv za Celokupno Lekarstvo; North Macedonia: Prilozi. There was no medical journal from Montenegro indexed in the Pubmed/MEDLINE. The following search strategy was used for each country separately to implement inclusion and exclusion criteria and select the pool of clinical/observational studies for further analysis: (“Title of the journal 1” [All Fields] OR „title of the journal 2“ [All Fields] OR ...) AND ((“observational study”[Publication Type] OR “observational studies as the topic”[MeSH Terms] OR “observational study”[All Fields]) OR (“clinical trial”[Publication Type] OR “clinical trials as the topic”[MeSH Terms] OR “clinical trial”[All Fields])) Filters: Free full text. The data for Bosnia and Herzegovina was used from our previous study published earlier this year (12). The size of the study sample for clinical/observational studies (n=43) was calculated based on the following assumptions: rate of inappropriate research design 0.5 (13) and width of the 95% confidence interval ± 0.15. The formula n = (1.96)2 x 4*p*(1-p)/d2 was used for the calculation, where „n“ is the sample size, „p“ probability of inappropriate research design, and „d“ width of the confidence interval (14). Since the studies retrieved by the abovementioned search strategy were numbered orderly in the PubMed database, the study sample of 43 studies was extracted by simple randomization technique, activating for 43 times random number generator in Excel, using formulas RANDBETWEEN (1;202) for Croatia and RANDBETWEEN (1;318) for Serbia. As already mentioned, the sample for Bosnia and Herzegovina was used from our previous study, and all retrieved studies were used for Slovenia and North Macedonia since their numbers were lower than 43. The extracted clinical/observational studies were analyzed for common errors in design and statistics, as earlier described in the literature (searching for sample size calculations, randomization techniques, assumptions for statistical tests, intention-to-treat analysis, etc.) (15). For the purpose of analysis of the extracted studies, the checklist with 9 questions was prepared (Table 2). The details of the analysis of the extracted studies are shown in Tables 1 and 2. The number of citations for each study was taken from Google Scholar after performing a search with the study title. The results were described by rates and percentages when categorical, and by means, standard deviations, medians, and interquartile ranges, if continuous. The normality of the data distribution was checked by the Kolmogorov-Smirnov test, and if not achieved, Kruskal-Wallis non-parametric analysis of variance was used for comparisons among the countries. Values of categorical variables were compared among the Ex-Yugoslav countries by the Chi-square test or by Fisher’s exact test if assumptions for the Chi-square test were not met. Association between the number of satisfied methodological criteria and journal impact factor, the number of citations or time elapsed from the publication date was tested by linear regression. The maximum acceptable probability of the null hypothesis was set at 0.05. All calculations were performed by the SPSS statistical program, version 18.

Table 2. Results of the methodological evaluations. * probability of null hypothesis calculated by Chi-square test or Fisher’s exact test (if frequencies less than five) for categorical data, and by Kruskal-Wallis non-parametric analysis of variance for continuous data.

	Slovenia	Croatia	Bosnia and Herzegovina	Serbia	p*
Failure to specify the inclusion and exclusion criteria?	6 (18%)	4 (9%)	7 (16%)	4 (9%)	0.595
Failure to determine and report the error of your measurement methods?	11 (32%)	19 (44%)	18 (41%)	26 (60%)	0.006
Failure to specify the exact statistical assumptions made in the analysis?	25 (74%)	22 (51%)	26 (59%)	37 (86%)	0.000
Failure to perform sample size analysis before the study begins?	30 (88%)	37 (86%)	39 (89%)	42 (98%)	0.215
Failure to implement adequate bias control measures?	16 (47%)	21 (49%)	26 (59%)	31 (72%)	0.006
Failure to vigorously recruit and retain subjects?	0 (0%)	1 (2%)	4 (10%)	0 (0%)	0.077
Failure to examine for normality of the data?	25 (74%)	21 (49%)	23 (52%)	38 (88%)	0.000
Failure to report missing data, dropped subjects and use of an intention to treat analysis?	4 (12%)	11 (26%)	32 (73%)	31 (72%)	0.000
Failure to point out the weaknesses of your own study?	14 (41%)	12 (28%)	27 (61%)	31 (72%)	0.000
Number of satisfied criteria per study: mean, standard deviation, median, interquartile range	5.1 ± 1.7, 5.0, [2.0]	5.5 ± 1.5, 6.0, [2.0]	3.6 ± 1.8, 3.0, [3.0]	3.4 ± 1.1, 3.0, [1.0]	0.000
Number of studies that satisfied more than 4 criteria	22 (65%)	32 (74%)	13 (30%)	7 (16%)	0.000

Table 1. Characteristics of the study samples per country. *Slovenia: Acta Dermatovenerologica Alpina, Pannonica, et Adriatica, Radiology and Oncology, and Zdravstveno Varstvo; Croatia: Acta Medico-historica Adriatica, Acta Dermatovenerologica Croatica, Biochemia Medica, Acta Pharmaceutica (Zagreb, Croatia), Croatian Medical Journal, Acta Clinica Croatica, Psychiatria Danubina and Arhiv za Higijenu Rada i Toksikologiju; Bosnia and Herzegovina: Acta Informatica Medica, Materia Socio-Medica, Medical Arhives, Acta Medica Academica, Medicinski Glasnik and Bosnian Journal of Basic Medical Sciences; Serbia: Acta Chirurgica Iugoslavica, Medicinski Pregled, Vojnosanitetski Pregled, and Srpski Arhiv za Celokupno Lekarstvo (the four Serbian journals were indexed in Pubmed/MEDLINE before year 2017, and are not currently indexed). ** standard deviation *** interquartile range **** probability of null hypothesis calculated by Kruskal-Wallis non-parametric analysis of variance.

	Slovenia	Croatia	Bosnia and Herzegovina	Serbia	p****
Number of medical journals publishing clinical studies and indexed in MEDLINE*	3	8	6	4	-
Number of retrieved studies	34	202	149	318	-
Size of the evaluated random sample	34	43	44	43	-
Years passed from a study publication (mean, SD**, median, IQR***)	5.3 ± 4.0, 3.5 [6.3]	10.1 ± 6.7, 7.0 [11.0]	4.7 ± 3.2, 4.0 [3.8]	10.0 ± 4.8, 8.0 [9.0]	0.000
Number of citations (mean, SD**, median, IQR***)	8.4 ± 11.7, 4.5 [10.3]	11.2 ± 18.2, 6.0 [7.0]	1.8 ± 1.9, 1.0 [2.8]	2.9 ± 2.5, 2.0 [4.0]	0.000
Impact factor (mean, SD**, median, IQR***)	1.8 ± 1.0, 2.2 [1.9]	1.3 ± 0.9, 1.0 [1.0]	0.9 ± 0.5, 0.9 [1.0]	0.2 ± 0.1, 0.2 [0.2]	0.000
Percent of foreign studies in the sample	59%	44%	52%	0%	-

4. RESULTS

From the six ex-Yugoslav countries evaluation of methodological errors in clinical studies published by medical journals indexed in Pubmed/MEDLINE database was possible only in four; Montenegro does not have a medical journal indexed in MEDLINE that publishes clinical studies, and North Macedonia has one (”Prilozi“), from which in total abstracts of 18 studies were retrieved that fulfilled inclusion criteria (16). However, the full text of these 18 studies was not available through the Pubmed/MEDLINE, or elsewhere on the Internet (the articles were not assigned with DOI numbers, additionally precluding a finding of full texts). Medical journals indexed in MEDLINE, number of retrieved studies, size of random samples, number of citations, years elapsed from the publication, impact factors of the journals in the year of publication, and percent of the samples with authors from foreign countries are shown in Table 1. Characteristics of the study samples per country. Number and percent of the evaluated studies that failed to meet each of the methodological criteria tested, the number of the evaluated criteria not satisfied per country, and the number of studies that satisfied more than 4 criteria per country is presented in Table 2. References of the analyzed studies from Bosnia and Herzegovina could be found in our previous publication (12), while the references of the studies analyzed from Croatian (17-59), Slovenian (60-92) and Serbian medical journals (93-135) are given on the list at the end of this article.

When the explanatory potential of journal impact factor, a number of citations and time elapsed from publication regarding the number of methodological criteria satisfied per study was tested by linear regression, it was not significant in samples of studies published in Slovenia, Bosnia and Herzegovina and Serbia. However, the methodological quality of studies published in Croatian medical journals was directly related to the impact factor of the journals (F=16.638, p=0.000, R2=0.289) and inversely with the time elapsed from the publication (F=6.404, p=0.015, R2=0.135). Each additional year of the time elapsed from the publication date decreased the number of satisfied methodological criteria for 0.084, while every additional unit of impact factor increased the number of satisfied criteria for further 0.895 points.

5. DISCUSSION

Scientific research in medicine is the process of a systematic study of well-defined aspects of physical, mental, and social well-being (1). The scientific research process has several components. These are the identification of the main research questions, the selection of a scientific approach, study design and data collection, data analysis, and reporting on achievements. Writing a paper is a tedious job, however, following the established rules that work not only becomes much simpler, but also more accessible, which often results in the birth of the desire of researchers to write an article. Knowledge of the principles established by the process of scientific research demystifies the process. Decomposition process research into simpler trying to animate all those who can contribute to the advancement of medical science. According to the material that they work on, biomedical journals can be divided into four groups (1): a) Narrowly specialized journals; b) General medical journals; c) Classic journals; and d) Primary scientific journals (1).

Narrowly specialized journals commonly deal with materials from a more narrow area of research. They are usually oriented towards one segment of medicine, very often this could be a specific organ or disease. These journals usually posses archive features, they are carefully prepared, strictly reviewed and of high scientific value. Nowadays most are issued on a quarter of a monthly basis, and some every month. The primary scientific journals are a part of the scientific literature and the main source of information. They are serial publications in which, for the first time, “the original research results and reviewed scientific articles” are published (1). Considering that there isn’t a monotonous criterion for the classification of these journals, it is hard to establish their exact quantity, but it is assumed that there are around 150,000 in existence now. According to some authors, the number of journals is doubled every 10 to 15 years, which amounts to an increase of 5% annually. The “Bradford distribution” law states that the most valuable information is published in a relatively small number of journals. According to this law 8,000 primary scientific journals, instead of the 30,000 being published now, would be quite sufficient (1). Many authors agree that the SCI database confirms this rule because in a selection of 4,000 to 5,000 journals it contains a majority of valuable information published worldwide annually.

There are two basic attributes of primary scientific journals: significance and value. We say that it is significant “if it efficaciously transmits scientific information, if it is readily available to scientists and if they read it.” (1).

Scientometrics is part of Scientology (the Science of Science) that analyzes scientific papers and their citation in the scientific journal selected sample (1). Name bibliometrics in the seventies was introduced to denote a quantitative study of the communication process using mathematical and statistical methods to books and other media of communication. Almost simultaneously, in the countries of the former Eastern Bloc was introduced Scientometrics name derived from the Russian language. More specifically, in 1969 was introduced the name scientometrics relating to scientific field that deals with the study of science as an information process by applying quantitative (statistical) method, and later Tibor Braun (who in 1977 established International Journal Scientometrics), introduces the name Scientometrics (1). Modern Scientometrics is based largely on the work of Derek J. de Solla Price and Eugene Garfield. Garfield founded ISI - Institute for Scientific Information is considered to be the father of scientometrics and methods of evaluation of scientific publications.

Research Methods scientific importance publications include qualitative and quantitative methods and computer analysis approach (1). Garfield has been striving to mathematical representation developed several factors that allow the assessment value and importance of scientific publications, including the most important Impact factor (IF) and the h-index. The most relevant indicators used in the evaluation of scientific research are: a) Impact factor, b ) Article citations, c) Journal citations, and d) The number and order of the authors, etc. The best measure of the significance of a journal is the ECHO factor, “which shows how much the published articles from a journal are cited” and tells us how much the journal is used and how important it is for scientists. The IMPACT factor shows “how much a scientific article in a journal is cited on average (1) and it is very similar to the echo factor, which shows how many articles are cited in a journal” (1). The value of a journal is a completely different question, that is to say, the role of information which is published in it for the sake of advancement in science.

The significance and value of primary scientific publications usually go together. It is an unusual occurrence that a small journal should publish an article of importance. That is why “the echo factor can be taken as an approximative indicator of the scientific value of a journal” (1). Croatian authors Zdravko Lackovic et al. estimated that worldwide 10,000 scientific and research publications are published daily from the field of medicine (1, 9). This sky-high number is a natural consequence of scientific progress, but also a result of the syndrome “publish or perish”, the quantitative instead of the qualitative evaluation of scientific papers. This is why every scientific milieu, no matter how big or small, tries to publish its own medical journal. Often, in the newly initiated journals, not enough attention is paid to the value of the published articles so that they are acceptable on an international level. The articles should be directed towards the worldwide scientific public and composed according to its rules. Another Croatian scientist Vlatko Silobrcic, in one of biomedical research and scientific information, described that “such a journal is the only one that makes sense of publishing because there is no other way in which a journal could be edited” (1). This is why financing a vast number of “scientific” journals, which do not satisfy international standards with their quality, is just an irrational way of emptying the science budget. In the majority of developed countries on every 1.000 researches, there are one to two or more journals that are in the SCI. In ex-Yugoslavia the number is 0.18. Also in developed countries on every 1,000 researchers there is one international journal, and in ex-Yugoslavia there is 1 (one) international journal approximately on every 2,000 to 2,500 researchers. Alongside the development of the economy in a country, the rate for the development of science depends also on the specific terms under which scientific research is conducted (1). The scientific productivity of our medical researchers is low. An average researcher publishes a work that is easily available to the international public every 5 years, and the ones from university do so every three to four years. Every 1,000 doctors publish six or seven, and those outside clinics and institute only 1.1 international clinical papers annually. These statistics are very poor indeed since according to the SCI we are lagging behind Austria, approximately 70 times (1). Database Pubmed/Medline as the largest source of deposited biomedical journals contains about 5500 medical journals (9). Before wartime in ex-Yugoslav countries in Pubmed/Medline were deposited several times more than today (9). In Pubmed/Medline have only 22: (Slovenia-3, Croatia-8, Bosnia and Herzegovina-6, Serbia-4 and North Macedonia-1), which we were included in our meta-analysis.

Our study showed that there are significant differences in methodological quality and the scientometric aspect of clinical research studies published in ex-Yugoslav countries. The studies published in two countries that are now member states of the European Union (Slovenia and Croatia) have significantly higher citation rates, impact factor, and methodological quality scores than studies from other countries. While publications from Croatia show clear improvement trend throughout the last two decades, which is visible also in the last 10 years in Slovenia and Bosnia and Herzegovina, quality of publications from journals published in Serbia was stagnating in the same period, probably being associated with removal of four Serbian journals from the MEDLINE.

There is a striking difference in the percent of studies from foreign countries published in medical journals from the region. While almost half of all studies published in medical journals from Slovenia, Croatia, and Bosnia and Herzegovina came from countries out of the region of ex-Yugoslav countries, the studies published in Serbian journals were exclusively from domestic authors. This is a significant factor that contributed to lower citation rates of studies published in Serbian journals and therefore lower impact factors since collaborative studies and international collaboration in general increase the chances of multiple citing of a published study (136).

Interestingly, there are also some similarities among the countries: inclusion and exclusion criteria were omitted in only a few studies, and almost all studies from all evaluated countries had very low dropout rate; on the other hand, sample size calculation was omitted in the great majority of studies, regardless of a country where they were published. Since sufficient power of a clinical study is crucial for avoidance of false-negative results (137), significant efforts should be made in the future to spread the knowledge and skills of calculating sample size and power of a study for all types of research design. Another significant problem in all analyzed countries was the frequent omission of checking whether assumptions of statistical tests were met by a study data, and whether the data were normally distributed, which should have a determined type of statistical tests that could be used for data analysis. Using a statistical test when its assumptions were not met directly gives erroneous results, and all their consequent interpretations would be unsubstantiated and most probably wrong. It is a pity that high-quality data, often collected by expensive and elaborious methods, are then processed inappropriately leading to incorrect conclusions (138). Again, in order to prevent waste of precious resources and get the best out of our research, it is necessary to invest much more in training clinical researchers with statistical skills in close future (139-140). The main limitation of our study is the potential for measuring bias since only two authors from two of the ex-Yugoslav countries were evaluating all retrieved studies. Besides, the method of measuring citation rates may have been unreliable, since lack of funding of this study precluded the use of specialized and validated citation software or paid services of university or national libraries.

6. CONCLUSION

Our study pointed to significant differences in methodological quality and scientometric characteristics of clinical research published in medical journals of ex-Yugoslav countries. Future investments in the training of clinical researchers in statistics and research design, as well as more rigorous editorial practices, are necessary to mitigate these differences and guide further progress in clinical research of all ex-Yugoslav countries.

Author’s contribution:

Both authors were included in all steps of preparation of this article. Final proof reading was made by both authors.

Conflicts of Interest:

None declared.

Financial support and sponsorchip:

Nil.