Abstract
Introduction:
The source of scientific information, methods for their evaluation, and methodology of their use are critical for serious scientific research and publishing of the scientific research results. Certain methodological principles should be inexcusably followed when designing clinical or observational research to avoid bias and presentation of results that do not reflect the truth about the phenomenon that is the object of the study.
Aim:
The aim of this study was to compare the methodological quality of clinical trials and observational studies published in medical journals from ex-Yugoslav countries indexed in Web of Science (WoS) and Pubmed/MEDLINE.
Methods:
Clinical studies published in medical journals of ex-Yugoslav countries were retrieved from the WoS and Pubmed database, and the sample for analysis was randomly chosen from the retrieved publications. The rate of the most common errors in the design of clinical/observational studies was established by a careful reading of the sampled publications and their checking against predefined criteria.
Results:
Number and percent of the evaluated studies that failed to meet each of the methodological criteria tested, number of the evaluated criteria not satisfied per database and number of studies that satisfied more than 4 criteria were analyzed per database. When explanatory potential of journal impact factor, number of citations, time elapsed from publication and a database where a journal is referred were tested by linear regression in regard to the number of methodological criteria satisfied per study, the linear regression model was obtained by backward deletion method and achieved R2 adjusted of 0.166 (F=13.827, df1 = 2, df2 = 127, p=0.000). The methodological quality of studies was directly related to impact factor of the journals (B = 0.976, 95% confidence interval 0.539 – 1.413, p=0.000) and inversely with the database where a journal is referred (B =–0.444, 95% confidence interval–0.824 – -0.064, p = 0.022). Each additional unit of impact factor increased number of satisfied methodological criteria for about 1, while referring a journal only in WoS decreased number of satisfied criteria for 0.45 points in comparison with journals referred in both WoS and Pubmed/MEDLINE, and for 0.9 points in comparison to journals referred only in MEDLINE.
Conclusion:
Methodological and scientometric quality of clinical studies published in medical journals from ex-Yugoslav region varies significantly, and the variations are higher in journals referenced only in WoS than in journals referenced in Pubmed/MEDLINE only, or in both Pubmed and Web of Science databases.
Keywords: Methodological errors, Clinical studies, Research design, Statistical errors
1. INTRODUCTION
A scientific research is a process that has several distinct components.(1) These are: To identify the key research questions, choice of scientific approach for the study and data collection, data analysis, and reporting on results (2) Science and technology play a key role in the development of modern society and scientific research, and if they stand on the ethical principles, they can certainly provide answers to others to remix, tweak, and build upon the previous work non-commercially, as long as the author is credited and the new creations are licensed under the identical terms (2). The production and exchange of knowledge are important issues of human existence that are reflected in relevant scientific communication, which is established and implemented by scientific publications (1). The reliability and soundness of the scientific knowledge of each scientist and researcher should be important for him/her and for the professional community to which it belongs. In this way, the doors would potentially turn open for their successful scientific and possible academic career.
The source of scientific information, methods for their evaluation, and methodology of their use are critical for serious scientific research and publishing of the scientific research results (1). The society normally imposes to science some rules of conduct and rules of the game. But scientific knowledge still largely depends on the actions that are at least in the initial phase and basically individual. It generally depends, to a large extent, on the creativity, skills, and talents of individuals. Creativity and critical thinking are just some of the essential characteristics of the scientific research process (2). From the standpoint of content, in order to distinguish papers with scientific ambitions from those which are called professional, we must recognize some of their basic characteristics.
Scientometrics is part of Scientology (Science of sciences) that analyzes scientific publications and their citation in the selected sample in scientific journals (1-3). Scientometrics is the science of measuring and analyzing science using qualitative, quantitative and computational approaches (2).
An important way of systematically reviewing studies is through meta-analysis (1). Meta-analysis is a statistical and analytical method that combines and synthesizes various mutually independent studies and integrates their results into a single, common result. If well designed and properly implemented, it can be a very powerful tool for proving hypotheses. It is based on strictly established mathematical and statistical principles for critical analysis of medical data. If the results are obtained by proper meta-analysis supervised by experts, they are considered valid and there is no need for further testing (2).
Meta-analysis became indispensable in understanding a large collection of raw data or literature that is sometimes contradictory, inconsistent, and unclear on a topic, and in understanding the true importance of statistical results when addressing a scientific topic, such as efficiency.
Scientometric quality of articles published in medical journals is a function of the quality of the journal itself, but also of the major database where the journal is referenced. It is well known that quality of the articles also vary over time (1), underlining its multifactorial nature. Although Web of Science (WoS) covers the most cited articles published in the journals with the highest impact factor (2), is also hosts a multitude of low impact factor journals which publish articles with very low scientometric rank. The latter is especially true for journals coming from certain geographical regions with moderate research potentials. Another major database of medical journals, Pubmed/MEDLINE, also covers journals of various scientometric quality (3), but has somewhat different inclusion and exclusion criteria from WoS. In practice, it often happens in certain developing regions that some medical journals are covered only by WoS, some only by Pubmed/MEDLINE and other by both databases. Such situation creates different opportunities for the journals, since depending on which database is preferred by national scientific authorities (e.g. Ministry of Science) and included in scores for ranking national journals, regional authors of scientific papers will choose some of the journals, and ignore the others. Region that covers former Yugoslav countries is just one of the kind, and regional medical journals are differently valued by local authors depending on their reference status in WoS and Pubmed/MEDLINE. However, whether the methodological validity of the articles (i.e. their true quality) follows regional popularity or prestige of WoS or MEDLINE remains to be established.
2. AIM
The aim of our study was to compare methodological and scientometric characteristics of clinical trials or observational clinical studies published in journals from former Yugoslav countries and referenced in either WoS, Pubmed/MEDLINE or both databases.
3. METHODS
Our study was of the cross/sectional type. The studies were retrieved for analysis from the journals referred in PubMed/MEDLINE (4), Web of Science database - Master journal list (5), or in the both databases. The published articles were included into the study based on the following criteria: journal article, published in a journal issued in Slovenia, Croatia, Bosnia and Herzegovina, Serbia, Montenegro or North Macedonia, original clinical trial, original observational clinical study, and full text availability. The exclusion criteria were: articles published in a language other than English or Bosnian/Croatian/Serbian, review articles, animal studies, in vitro studies, modeling studies and case reports or case series. In order to create and employ search strategies, at first the following journals published in Ex-Yugoslav countries were identified: journals referred in Web of Science (WoS) and not in Pubmed/MEDLINE database (Journal of Medical Biochemistry, Acta Facultatis Medicae Naissensis, Acta Stomatologica Croatica, Alcoholism and Psychiatry Research, Signa Vitae, Onkologija, Balkan Journal of Medical Genetics, Montenegrin Journal of Sports Science and Medicine, Vojnosanitetski Pregled, and Srpski Arhiv za Celokupno Lekarstvo), journals referred only in Pubmed/MEDLINE database, and not in WoS (Acta medico-historica adriatica, Acta Informatica Medica, Materia Socio-Medica, Medical Archives, Acta Medica Academica and Medicinski Glasnik), and journals referred in both databases (Acta Dermatovenerologica Alpina, Pannonica, et Adriatica, Radiology and Oncology, Zdravstveno Varstvo, Acta Dermatovenerologica Croatica, Biochemia Medica, Acta Pharmaceutica (Zagreb, Croatia), Croatian Medical Journal, Acta Clinica Croatica, Psychiatria Danubina, Arhiv za Higijenu rada i Toksikologiju, Bosnian Journal of Basic Medical Sciences and Prilozi). The following search strategy was used for each country separately to implement inclusion and exclusion criteria and select clinical/observational studies for further analysis: ("title of the journal 1" [All Fields] OR „title of the journal 2“ [All Fields] OR ...) AND (("observational study"[Publication Type] OR "observational studies as topic"[MeSH Terms] OR "observational study"[All Fields]) OR ("clinical trial"[Publication Type] OR "clinical trials as topic"[MeSH Terms] OR "clinical trial"[All Fields])) Filters: Free full text. The data for journals referred in both WoS and PubMed were used from our previous study (6).
Size of the study sample in regard to the number of studies per database (n=43) was calculated on the basis of the following assumptions: rate of inappropriate research design 0.5 (7) and width of the 95% confidence interval ± 0.15. The formula n = (1.96)2 x 4*p*(1-p)/d2 was used for the calculation, where „n“ is the sample size, „p“ probability of inappropriate research design and „d“ width of the confidence interval (8). Since the studies retrieved were numbered orderly in the respective database, the study sample was extracted by simple randomization, using random number generator in Excel. The sample of studies published in journals referenced in both WoS and PubMed was used as analyzed in our previous study (6).
The extracted clinical/observational studies were analyzed for common errors in design and statistics, as earlier described in the literature (searching for data distribution, randomization techniques, inclusion and exclusion criteria, intention-to-treat analysis, etc.) (9). For the purpose of analysis of the extracted studies, the checklist with 9 questions was prepared (Table 2). Results of the analysis of the extracted studies are shown in the Tables 1 and 2. Number of citations for each study was taken from Google Scholar after performing search with the study title. The results were described by rates and percentages when categorical, and by means, standard deviations, medians and interquartile ranges, if continuous. Normality of the data distribution was checked by Kolmogorov-Smirnov test, and if not achieved, Kruskal-Wallis nonparametric analysis of variance was used for comparisons among the databases. Values of categorical variables were compared among the databases by Chi-square test or by Fisher’s exact test, if assumptions for the Chi-square test were not met. Association between number of satisfied methodological criteria and journal impact factor, number of citations or time elapsed from the publication date was tested by multiple linear regression. Maximum acceptable probability of null hypothesis was set at 0.05. All calculations were performed by SPSS statistical program, version 18.
Table 2. Results of the methodological evaluation. * probability of null hypothesis calculated by Chi-square test or Fisher’s exact test (if frequencies less than five) for categorical data, and by Kruskal-Wallis non-parametric analysis of variance for continuous data.
| Indexed only in WoS | Indexed only in Pubmed/MEDLINE | Indexed both in WoS and Pubmed/MEDLINE | p* | |
|---|---|---|---|---|
| Failure to specify the inclusion and exclusion criteria? | 23 (54%) | 7 (16%) | 4 (9%) | 0.000 |
| Failure to determine and report the error of your measurement methods? |
19 (44%) | 10 (23%) | 19 (44%) | 0.068 |
| Failure to specify the exact statistical assumptions made in the analysis? |
31 (72%) | 31 (72%) | 22 (51%) | 0.063 |
| Failure to perform sample size analysis before the study begins? |
39 (91%) | 39 (91%) | 37 (86%) | 0.725 |
| Failure to implement adequate bias control measures? | 31 (72%) | 23 (54%) | 21 (49%) | 0.069 |
| Failure to vigorously recruit and retain subjects? | 0 (0%) | 1 (2%) | 1 (2%) | 0.601 |
| Failure to examine for normality of the data? | 25 (58%) | 29 (67%) | 21 (49%) | 0.217 |
| Failure to report missing data, dropped subjects and use of an intention to treat analysis? |
38 (88%) | 25 (58%) | 11 (26%) | 0.000 |
| Failure to point out the weaknesses of your own study? | 26 (60%) | 24 (56%) | 12 (28%) | 0.005 |
| Number of satisfied criteria per study: mean, standard deviation, median, interquartile range | 3.6 ± 1.8, 3.0, [3.0] |
4.7 ± 2.0, 5.0, [3.0] |
5.5 ± 1.5, 6.0, [2.0] |
0.000 |
| Number of studies that satisfied more than 4 criteria | 12 (28%) | 23 (52%) | 32 (74%) | 0.000 |
Table 1. Characteristics of the study samples per country. * standard deviation, ** interquartile range, *** probability of null hypothesis calculated by Kruskal-Wallis non-parametric analysis of variance.
| Indexed only in WoS | Indexed only in Pubmed/MEDLINE | Indexed both in WoS and Pubmed/MEDLINE | p*** | |
|---|---|---|---|---|
| Number of medical journals publishing clinical studies | 10 | 6 | 12 | - |
| Number of retrieved studies | 523 | 91 | 285 | - |
| Size of the evaluated random sample | 43 | 43 | 43 | - |
| Years passed from a study publication (mean, SD*, median, IQR**) | 3.2 ± 2.6, 2.0 [4.0] |
4.3 ± 2.5, 4.0 [4.0] |
10.1 ± 6.7, 7.0 [11.0] |
0.000 |
| Number of citations (mean, SD*, median, IQR**) | 2.9 ± 4.0, 1.0 [5.0] |
8.5 ± 13.6, 4.0 [10.5] |
11.2 ± 18.2, 6.0 [7.0] |
0.008 |
| Impact factor (mean, SD*, median, IQR**) | 0.3 ± 0.4, 0.2 [0.2] |
0.6 ± 0.3, 0.6 [0.5] |
1.3 ± 0.9, 1.0 [1.0] |
0.000 |
| Percent of foreign studies in the sample | 34% | 49% | 35% | - |
4. RESULTS
There are ten biomedical journals from the six ex-Yugoslav countries referred only in WoS and not in Pubmed/MEDLINE database: Journal of Medical Biochemistry, Acta Facultatis Medicae Naissensis, Acta Stomatologica Croatica, Alcoholism and Psychiatry Research, Signa Vitae, Onkologija, Balkan Journal of Medical Genetics, Montenegrin Journal of Sports Science and Medicine, Vojnosanitetski pregled, and Srpski Arhiv za Celokupno Lekarstvo. Another six medical journals are referred only in Pubmed/MEDLINE database, and not in WoS (Acta Medico-historica Adriatica, Acta Informatica Medica, Materia Socio-Medica, Medical Archives, Acta Medica Academica and Medicinski Glasnik). Finally, both in WoS and MEDLINE are referred the following medical journals published in Ex-Yugoslav countries: Acta Dermatovenerologica Alpina, Pannonica, et Adriatica, Radiology and Oncology, Zdravstveno Varstvo, Acta Dermatovenerologica Croatica, Biochemia Medica, Acta Pharmaceutica (Zagreb, Croatia), Croatian Medical Journal, Acta Clinica Croatica, Psychiatria Danubina, Arhiv za Higijenu Rada i Toksikologiju, Bosnian Journal of Basic Medical Sciences and Prilozi.
Number of medical journals indexed in WoS, Pubmed/MEDLINE, or both databases, number of retrieved studies, size of random samples, number of citations, years elapsed from the publication, impact factors of the journals in the year of publication and percent of the samples with authors from foreign countries are shown in the Table 1.
Number and percent of the evaluated studies that failed to meet each of the methodological criteria tested, number of the evaluated criteria not satisfied per database and number of studies that satisfied more than 4 criteria per database are presented in the Table 2. References of the analyzed studies referred in both WoS and MEDLINE could be found in our previous publication (6), while the references of the studies covered only by WoS or only by Pubmed/MEDLINE (10-96) medical journals are given at the end of this article (Table 3 and 4). When explanatory potential of journal impact factor, number of citations, time elapsed from publication and a database where a journal is referred were tested by linear regression in regard to the number of methodological criteria satisfied per study, the linear regression model was obtained by backward deletion method and achieved R2 adjusted of 0.166 (F=13.827, df1 = 2, df2 = 127, p=0.000). The methodological quality of studies was directly related to impact factor of the journals (B = 0.976, 95% confidence interval 0.539 – 1.413, p=0.000) and inversely with the database where a journal is referred (B =–0.444, 95% confidence interval–0.824 – -0.064, p = 0.022). Each additional unit of impact factor increased number of satisfied methodological criteria for about 1, while referring a journal only in WoS decreased number of satisfied criteria for 0.45 points in comparison with journals referred in both WoS and MEDLINE, and for 0.9 points in comparison to journals referred only in Pubmed/MEDLINE.
Table 3. The list of articles published in the Journals referred in WoS only (first author and number in the cited list of references).
| Author’s of the Study | Failure to specify the inclusion and exclusion criteria | Failure to determine and report the error of your measurement methods |
Failure to specify the exact statistical assumptions made in the analysis |
Failure to perform sample size analysis before the study begins |
Failure to implement adequate bias control measures | Failure to vigorously recruit and retain subjects | Failure to examine for normality of the data | Failure to report missing data, dropped subjects and use of an intention to treat analysis |
Failure to point out the weaknesses of your own study |
Citation | IF of a journal |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Vilibić M, et al. (10) | No | Yes | No | No | Yes | No | No | No | No | 1 | 0.390 |
| Blažinović I, et al. (11) | No | NA | Yes | Yes | Yes | No | Yes | No | Yes | 0 | 0.390 |
| Jiang B, (12) | No | Yes | Yes | Yes | Yes | No | Yes | Yes | No | 3 | 1.593 |
| Sivaci R, et al. (14) | No | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | 0 | 0.622 |
| Zogović D , et al. (13) | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | No | 7 | 0.919 |
| de Almeida O, et al. (15) | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | 1 | 0.403 |
| Stefanović A, et al. (17) | No | No | Yes | Yes | Yes | No | No | Yes | No | 3 | 1.592 |
| Al-Zeidaneen AS, et al. (16) | No | Yes | Yes | Yes | Yes | No | Yes | Yes | No | 0 | 0.240 |
| Đorđević N, et al. (18) | Yes | NA | Yes | Yes | Yes | No | Yes | Yes | Yes | 0 | 0.210 |
| Živković VD, et al. (19) | Yes | NA | Yes | Yes | Yes | No | Yes | Yes | Yes | 0 | 0.200 |
| Kozomara S, et al. (20) | No | NA | Yes | Yes | Yes | No | Yes | Yes | Yes | 1 | 0.200 |
| Nešić I, et al. (21) | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | 0 | 0.240 |
| Gašpar M, et al. (22) | Yes | NA | Yes | Yes | Yes | No | Yes | Yes | Yes | 1 | 0.220 |
| Haliti F, et al. (23) | Yes | NA | Yes | Yes | Yes | No | Yes | Yes | Yes | 8 | 0.320 |
| Alajbeg IŽ, et al. (24) | No | No | Yes | Yes | No | No | No | No | No | 10 | 0.640 |
| Čelebić A, et al. (25) | No | NA | Yes | Yes | Yes | No | No | No | No | 1 | 0.640 |
| Muhić E, et al. (26) | No | No | Yes | Yes | No | No | No | Yes | Yes | 2 | 0.340 |
| Erceg T, et al. (27) | Yes | NA | Yes | Yes | Yes | No | Yes | Yes | No | 1 | 0.390 |
| Grizelj M, et al. (28) | Yes | Yes | No | Yes | Yes | No | No | Yes | Yes | 1 | 0.390 |
| Lotar Rihtarić M, et al. (29) | No | Yes | Yes | Yes | Yes | No | Yes | Yes | No | 5 | 0.390 |
| Sidlauskiene A, et al. (30) | No | No | No | Yes | No | No | No | Yes | No | 3 | 0.152 |
| Grbović V, et al. (31) | No | NA | No | Yes | No | No | No | Yes | Yes | 0 | 0.152 |
| Stefanović Z, et al. (32) | No | NA | Yes | Yes | Yes | No | No | Yes | Yes | 0 | 0.152 |
| Ivanović S, et al. (33) | No | No | No | No | Yes | No | No | Yes | Yes | 4 | 0.267 |
| Gajić M, et al. (34) | Yes | No | Yes | Yes | No | No | Yes | Yes | Yes | 5 | 0.267 |
| Kim JY, et al. (35) | Yes | NA | No | Yes | Yes | No | Yes | Yes | No | 0 | 0.170 |
| Karlović Z, et al. (36) | No | NA | No | Yes | No | No | No | Yes | No | 0 | 0.173 |
| Prkačin I, et al. (37) | Yes | NA | Yes | Yes | Yes | No | Yes | Yes | Yes | 4 | 0.126 |
| Kosmač N, et al. (38) | Yes | Yes | No | No | No | No | No | No | Yes | 0 | 0.136 |
| Marić N, et al. (39) | Yes | NA | Yes | Yes | NA | NA | Yes | Yes | Yes | 2 | 0.154 |
| Marciuš VŽ. et al. (40) | Yes | Yes | No | Yes | Yes | No | No | Yes | Yes | 0 | 0 |
| Kukovica MG, et al. (41) | No | NA | Yes | Yes | Yes | No | Yes | Yes | No | 0 | 0 |
| Škof E, et al. (42) | Yes | NA | Yes | Yes | Yes | No | Yes | Yes | Yes | 0 | 0 |
| Demir S, et al. (43) | Yes | Yes | Yes | Yes | Yes | NA | Yes | Yes | Yes | 0 | 0.692 |
| Terzic M, et al. (44) | No | Yes | Yes | Yes | Yes | No | yes | yes | No | 0 | 0.692 |
| Vachev TI, et al. (45) | No | Yes | No | Yes | No | No | No | Yes | Yes | 5 | 0.553 |
| Khabour OF, et al. (46) | No | Yes | No | No | No | No | No | Yes | Yes | 6 | 0.200 |
| Sobti RC, et al. (47) | Yes | Yes | No | Yes | Yes | No | No | Yes | Yes | 5 | 0.375 |
| Wertheimer V, et al. (48) | Yes | Yes | Yes | Yes | No | NA | Yes | Yes | No | 7 | 0 |
| Arazi H, et al. (49) | Yes | Yes | Yes | Yes | No | No | No | Yes | Yes | 6 | 0 |
| Muratović A, et al. (50) | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | No | 18 | 0 |
| Zhang Y, et al. (51) | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | 15 | 0 |
| Brás R, et al. (52) | Yes | No | Yes | Yes | Yes | No | Yes | Yes | No | 1 | 0 |
Table 4. The list of articles published in journals referred in Pubmed only (first author and number in the cited list of references).
| Author’s of the Study | Failure to specify the inclusion and exclusion criteria | Failure to determine and report the error of your measurement methods |
Failure to specify the exact statistical assumptions made in the analysis |
Failure to perform sample size analysis before the study begins |
Failure to implement adequate bias control measures | Failure to vigorously recruit and retain subjects | Failure to examine for normality of the data | Failure to report missing data, dropped subjects and use of an intention to treat analysis |
Failure to point out the weaknesses of your own study |
Citation | IF of a journal |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Duc NM, et al. (53) | Yes | NA | Yes | NA | NA | No | Yes | NA | No | 0 | 1.03 |
| Askari-Majdabadi H, et al. (54) | Yes | NA | Yes | Yes | Yes | No | Yes | No | No | 0 | 1.03 |
| Selmanovic S, et al. (55) | No | Yes | Yes | Yes | No | No | Yes | Yes | Yes | 11 | 0.76 |
| Gholipour Baradari A, et al. (56) | No | No | No | Yes | No | No | No | No | No | 6 | 0.76 |
| Lajqi N, et al. (57) | Yes | No | Yes | Yes | Yes | No | Yes | Yes | Yes | 14 | 0.90 |
| Berisha M, et al. (58) | No | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | 0 | 1.17 |
| Gojkov-Vukelic M, et al. (59) | No | No | Yes | Yes | Yes | No | Yes | Yes | Yes | 32 | 0.99 |
| Forootan M, et al. (60) | No | NA | Yes | Yes | No | No | Yes | No | No | 0 | 0.93 |
| Latsou D, et al. (61) | No | No | No | No | No | NA | No | No | No | 3 | 0.69 |
| Moghadam MP, et al. (62) | No | No | Yes | Yes | No | No | Yes | Yes | Yes | 4 | 0.78 |
| Jafarzadeh L, et al. (63) | No | NA | Yes | Yes | Yes | Yes | Yes | No | Yes | 13 | 0.86 |
| Jafari F, et al. (64) | No | No | Yes | Yes | Yes | No | Yes | Yes | Yes | 14 | 0.63 |
| Petrovic M, et al. (65) | No | Yes | Yes | Yes | Yes | Not clear | Yes | Yes | Yes | 10 | 0.25 |
| Garousi S, et al. (66) | Yes | No | Yes | Yes | No | Not clear | Yes | Yes | No | 5 | 0.25 |
| Rohani F, et al. (67) | No | NA | Yes | Yes | Yes | No | Yes | Yes | Yes | 0 | 0.76 |
| Hoxha A, et al. (68) | No | NA | Yes | Yes | No | No | Yes | Yes | No | 3 | 0.76 |
| Torkaman A, et al. (69) | No | NA | Yes | Yes | Yes | No | Yes | No | Yes | 4 | 0.61 |
| Vanis-Vatrenjak S, et al. (70) | No | NA | Yes | Yes | Yes | No | Yes | No | Yes | 2 | 0.61 |
| Haxhibeqiri-Karabdic I. (71) | No | NA | No | Yes | NA | No | No | No | Yes | 32 | 0.65 |
| Djedovic M, et al. (72) | No | Yes | Yes | Yes | Yes | No | Yes | No | Yes | 4 | 0.66 |
| Rahmanovic E, et al. (73) | No | NA | No | Yes | No | No | Yes | Yes | Yes | 3 | 0.55 |
| Zonić-Imamović M, et al. (74) | No | NA | Yes | Yes | No | No | No | Yes | Yes | 0 | 0.17 |
| Bajrić B, et al. (75) | No | No | No | Yes | Yes | No | No | No | No | 2 | 0.17 |
| Račić M, et al. (76) | No | No | Yes | Yes | NA | No | Yes | Yes | No | 1 | 0.27 |
| Godinjak A, et al. (77) | No | No | Yes | Yes | NA | No | Yes | No | No | 47 | 0.14 |
| Meneses Calderón J, et al. (78) | No | NA | Yes | Yes | No | No | Yes | Yes | No | 2 | 0.14 |
| Krstović-Spremo V, et al. (79) | No | No | Yes | No | No | No | Yes | No | No | 11 | 0.27 |
| Loghmani A, et al. (80) | No | No | No | No | No | No | No | No | No | 66 | 0.27 |
| Iglica A, et al. (81) | No | NA | No | Yes | Yes | No | Yes | No | Yes | 0 | 0.38 |
| Rakanović-Todić M, et al. (82) | No | NA | Yes | Yes | Yes | No | Yes | Yes | Yes | 0 | 0.38 |
| Vegar-Zubović S, et al. (83) | No | NA | No | Yes | No | No | No | Yes | Yes | 5 | 0.29 |
| Okanović A, et al. (84) | No | Yes | No | Yes | Yes | No | No | Yes | No | 33 | 0.28 |
| Mujarić E, et al. (85) | No | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | 6 | 0.66 |
| Mehmedović A, et al. (86) | No | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | 6 | 0.53 |
| Gholipour Baradari A, et al. (87) | No | Yes | Yes | Yes | No | No | Yes | No | No | 6 | 0.28 |
| Stevanović S, et al. (88) | No | No | Yes | Yes | Yes | No | Yes | NA | No | 13 | 0.12 |
| Nouraei SM, et al. (89) | No | No | No | No | No | No | No | No | No | 4 | 0.61 |
| Ðug H, et al. (90) | Yes | Yes | Yes | Yes | NA | No | No | Yes | Yes | 2 | 0.29 |
| Asheghan M, et al. (91) | No | NA | Yes | Yes | Yes | No | Yes | Yes | Yes | 2 | 0.78 |
| Stanetić K, et al. (92) | Yes | No | No | Yes | Yes | No | No | Yes | No | 0 | 0.17 |
| Alimian M, et al. (93) | No | Yes | No | Yes | Yes | No | No | Yes | Yes | 5 | 0.65 |
| Alkhaldi HM, et al. (94) | Yes | NA | Yes | Yes | Yes | No | NA | Yes | Yes | 2 | 0.61 |
| Esparza M, et al. (95) | No | NA | Yes | Yes | Yes | No | NA | Yes | No | 0 | 0.17 |
| Secik Arkin F, et al. (96) | No | No | No | Yes | No | No | No | NA | No | 0 | 1.03 |
Interesting facts are shown in Table 3 and 4 (in the last two columns) - that papers with numbers 50 (Muratovic et al.) and 51 (Zhang et al.) have the highest number of citations but papers are published in WoS indexed journals with IF 0. Or, the fact is that the most cited papers published in Pubmed journals (Loghman et al. - 66 citations, Haxhibekiri-Karabdic et al. - 32 citations), were published in journals with relatively low IF (0,27 and 0,65), but Duc et al. has not citations, and he published paper in the journal with higher IF than those of 95% of journals where the analyzed studies in our sample were published. Also, citation of the paper from Pubmed list, published by author Gojkov-Vekelic et al. in journal indexed in Pubmed has higher number of citations, and journal in which paper is published has higher IF, but paper belongs to Dental medicine scientific area, not to areas of medicine, which are more important or attractive, or have more influence in medical sciences - within the scientific medical subdisciplines, like Internal medicine or Surgery, etc. In this study we did not compare authors end their achieved, e.g. number of citations, or IF scores of the journals, for how long time journals existed, in which country journals are printed, or how long time journals are deposited in WoS or Pubmed/MEDLINE databases, or to which scientific fields journals belong, or with score of Google Scholar indexes of authors which are included in our meta-analysis. It will be, probably, scope of our next analysis in the future. Generally, we proved that number of citations and IF scores in our analysis was not strongly connected with belonging of the journals to WoS or Pubmed/MEDLINE databases.
5. DISCUSSION
Scientometrics with its various indices is a reliable method for evaluation of scientific development (1, 2, 97). Name bibliometrics in the seventies was introduced to denote a quantitative study of the communication process using mathematical and statistical methods to books and other media of communication (6). More specifically, in 1969 was introduced the name scientometrics relating to scientific field that deals with the study of science as an information process by applying quantitative (statistical) method, and later Tibor Braun (founder of the International Journal of Scientometrics), introduced the name Scientometrics (97). Modern Scientometrics is based largely on the work of Derek J. de Solla Price and Eugene Garfield, founder of Institute for Scientific Information (ISI), named as the father of Scientometrics and methods of evaluation of scientific publications (1, 97, 102). Garfield developed several factors that allow the assessment value and importance of scientific publications, including the most important impact factor (IF) and the H-index. IF is the number of citations of articles published in the journal during the previous two years divided by the total number of articles published in the journal during the same period (97, 102).
In the August 2005, Jorge Hirsch introduced a new indicator for quantifying the research output of scientists (1, 97). Hirsch’s so-called H index was proposed as an alternative to other bibliometric indicators - such as the number of publications, the average number of citations and the sum of all citations (97) - and is defined as follows: “A scientist has index h if h of his or her Np papers have at least h citations each and the other (Np - h) papers have ≤ h citations each” (102). All papers by a scientist that have at least h citations are called the “Hirsch core” (13). An H-index of 5 means that a scientist has published five papers that each have at least five citations (97). An H-index of 0 does not inevitably indicate that a scientist has been completely inactive: he or she might have already published a number of papers, but if none of the papers was cited at least once, the H-index is 0 (97). H-index is an index that attempts to measure the productivity and impact of published work of scientists (the index is based on the basis of the most cited papers and the number of citations that papers received in other publications). One criticism is that the excess citations, i.e., all citations which exceed h for a given publication do not have an effect. Therefore Egghe (97, 102) proposed the g-index which is given by the largest number g of papers which have received at least g citations on average (It was argued recently that the g-index is a measure of a researcher’s specific impact (i.e., impact per paper) as much as it is a measure of overall impact) (97). The main difference between the g-index and the H-index is that the former penalizes consistency of impact whereas the latter rewards such consistency (it is concluded that the H-index is a better bibliometric tool than is the g-index) (1, 2).
For all authors is very important question for thinking before making decisions about his/her written paper: How to choose an academic journal for their research works? (97, 98). "Find out information about the relative quality of a journal either by using the programs Publish or Perish and Scopus to obtain data on the impact factor and citation rate of journals in your area of interest; by accessing Web of Knowledge or Science through your institution to obtain rankings of the journals in a particular area; or by visiting the journal’s website to find out information about the journal in question" (98, 99).
Our study showed that studies published in journals from former Yugoslav countries were of the highest methodological quality if referenced in both WoS and Pubmed/MEDLINE, and of the lowest if referenced only in WoS. Especially constructive self-critique, better definition of inclusion and exclusion criteria and missing data were more prevalent in studies published at journals covered by Pubmed/MEDLINE. Methodological quality was correlated with scientometric quality, since impact factor of the journals and number of citations were also higher for articles covered by Pubmed/MEDLINE or both WoS and MEDLINE in comparison to those referenced only in WoS.
This article represents a continuation of consideration on a very important and methodologically significant problem within the scientific field - Scientometry, especially in the field of biomedical sciences, about which the experiences in written form are very scarce. Few authors in the field of medicine in the former Yugoslavia have dealt with this aspect (several articles in this field have been published by the authors: Lackovic, Rumbolt, Srica, Marusic, mostly descriptively without meta-analysis), and written experiences are insufficient even globally in medical sciences. Prior to its disintegration in 1991, the former Yugoslavia had over 22 million inhabitants in 6 independent states and over 15 medical and dental faculties, and more than 15,000 teachers elected in academic titles in over 40 medical scientific disciplines. Also, more than 100 indexed biomedical journals have been continuously published.
These indicators are presented in more detail in several articles by both authors of this article and readers can find more about this topic from them (99-102). This study, with its results of applied meta-analyzes and a unique and specific approach, provides several important conclusions, but also messages and recommendations to readers, not only informative, but also educational, for authors who write articles, for reviewers who evaluate articles before publication, and especially for journal evaluators in their indexing application in index and citation databases.
The idea for this study arose after the First and the Second SWEPs (Mediterannean Seminar on Science Writing, Editing and Publishing), held in Sarajevo in 2016 and 2018, and after we in a previous study (103-105), were analyzing errors in statistical analysis of data in articles indexed in WoS journals, with IF ranging from 2.5 to 4.5, and found a large number of methodological errors. We came to the conclusion that the editors of journals and their associates, as well as mentors of master’s and doctoral theses (because a large number of published papers were taken from the content of these defended theses) had insufficient knowledge in the field of Medical Statistics. A special case are expert evaluators of potential journals for admission to index databases such as WoS, Medline, Scopus, etc. The experience we have with applications in these three databases indicate that expert evaluators take as leading criteria: scope of the journal, regional character of the journal, quality of the language used (they prefer US rather than UK English), technical features, quality of the journal web page, content of the Instructions for authors with COPE Guidelines and based on the ICMJE criteria (http://www.icmje.org/recommendations/browse/roles-and-responsibilities/defining-the-role-of-authors-and-contributors.html), "Sarajevo Declaration on Integrity and Visibility of Scholarly Journals" (104). other rules, etc.
In this article, we cannot analyze in detail the facts that are the arguments for our allegations, but we refer readers to the last two columns of Tables 3 and 4, which list the citations of articles and Impact Factors of journals in which articles are published and the correlation between them, and which indicate to and argument the above allegations. Let’s say for example: what is the correlation between WoS and PubMed indexing of the journal and Impact factors, the correlation between the length of existence (age) of the journal and Impact factors, the correlation between the importance of the topic of the article and Impact factors, etc. A special case for analysis is the role and significance of the effects of the List of Predatory Journals by American librarian Jeffrey Beall, whose list of Potential Predatory journals is cited by many authors, based on his own criteria that none of the world’s scientific and academic institutions have analyzed or accredited, but which could be officially used. His list has provoked a storm of outrage among thousands of publishers and editors of journals, who have been put in a position to be scientifically belittled by his criteria and list, and many authors to avoid them as potential journals in which to publish their articles. The authors which are quoting Beall and his list did not use a scientific method similar to ours at all - a method of meta-analyzing articles from his list in which they could confirm Beall’s assessments and the content and quality of articles from the list of predatory journals. An illustrative article by Refat Aljumili on a serious and critical approach has revealed what we have stated in this article: "The story of „Beall’s List“ started probably in early or late 2010 when a guy called Jeffrey Beall - a librarian at Auraria Library, University of Colorado, in Denver, Colorado - came up with a blog „Scholarly Open Access“, as well as a list of questionable journals and publishers, or as Jeffrey Beall likes to call it „Potential, possible or probable predatory scholarly open-access publishers“, and gave himself the right to ward academic scholarly academic publication" (98).
Shortly after this list established, Jeffrey Beall added many open-access publishers to it and continued to update it regularly - by adding to the list and removing from it - and introduced many authors and researchers to the assumption that Open Access Journals (OAJ) are essentially “Predatory publishers and low-quality journals”. Sadly, few authors, academics, or researchers who read or heard about “Beall’s List” without much previous knowledge of the academic publication situation in general and without any prior knowledge about Jeffrey Beall’s background and intentions in particular came away believing that “Beall’s list” is a recognized authority in evaluating scholarly journals (98). "Well, the truth is “Beall’s list” has no affiliation to any governing body or organization accredited to scholarly publishing, and has no legal or academic value. If you follow some of Beall’s work on his blog, and it makes no sense whatsoever! His official web blog exposed the truth about Jeffrey Beall, particularly Walt Crawford’s 2014 article “The Sad Case of Jeffrey Beall - Case and Insights”, which provides a very detailed history about OAJs and directly addresses some of the broader issues with “Beall’s list”., (available on: http://citesandinsights.info/civ14i4.pdf). (98, 99).
Additionally, two examples of arguments for statements in this article are Scopus indexed journals: at the https://blog.cabells.com/category/predatory-publishing/ we found interesting fact that 81 Vietnamese authors who published their articles in the 2019 IJEAT ("International Journal of Engineering and Advanced Technology") and were rewarded, were also indicated as predatory, but journal was indexed in Scopus in 2019 (https://www.scopus.com/sourceid/21100899502) (106). According to Scopus, in 2018, there were only 280 documents from International Journal of Engineering and Advanced Technology indexed in Scopus content coverage. Surprisingly, in 2019, number of documents, indexed in Scopus, raised up to 4455. It is remarkable that no external citations from these indexed documents were noticed. Eventually, coverage discontinued in Scopus in 2019 (Figure 1). We believe that the Content Selection and Advisory Board of Scopus should consider their workflow again when evaluating new applicants and indexing massively documents from new members to Scopus database like aforementioned example.
Figure 1. Scopus discontinued indexing "International Journal of Engineering and Advanced Technology" in their content coverage in 2019.
Second case is journal "Folia Medica Facultatis Medicinae Universitatis Saraeviensis" (http://www.foliamedica.mf.unsa.ba/index.php/FM), (in journal has commonly been published papers of Medical faculty in Sarajevo teachers), which was also accepted to Scopus several years ago without serious evaluation (re-established after more than 20 years of brake). It is stated that its last issue is printed in March 2019, and journal is signed as Croatian journal which belongs to University of Zagreb, Croatia. But the journal is published in Sarajevo (Bosnia and Herzegovina), and its h-index is 1 (107).
On the other side, the journal Materia Socio-Medica, one of the oldest journals on the Public Health in Europe (established in 1978) was rejected by Scopus several times, while in this study it has papers with high IF (papers 60 to 64, with Ifs from 0.63 to 0.93), better than 50% of IFs of WoS journals in our study.
The most important facts is that Editors and reviewers need to approach the evaluation of manuscripts submitted to journals more responsibly. Editors and reviewers should not reject articles without arguments, nor accept articles without checking that the submitted articles are written in accordance with the appropriate elements of the methodology that guarantee impartiality and proper application of statistics, all in order to reach the scientific truth in medicine (97, 102, 108, 109).
This is reason why studies like this one need to be made much more frequently in the praxis, especially for evaluations of the journals quality.
Limitation of the study: Our study had several limitations: a) only two investigators from two of the six former Yugoslav countries rated all articles, which created an opportunity for rating bias; b) the criteria for methodological quality were based on just one reference, therefore some of important methodological qualities could have been missed; and c) gold standard for overall methodological quality of clinical trials and observational clinical studies is still lacking, therefore our results should be taken with a reserve.
6. CONCLUSION
Methodological and scientometric quality of clinical studies published in medical journals from ex-Yugoslav region varies significantly, and the variations are higher in journals referenced only in WoS than in journals referenced in Pubmed/MEDLINE only, or in both Pubmed and Web of Science databases. Editors and reviewers of the submitted papers need to keep respect of the journal and check all methodological elements in assessment of the quality of the submitted papers, especially of the used correct statistical analysis and presentation of the results, trying to keep the scientific truth in medicine.
Author’s contribution:
Both authors were included in all steps of preparation of this article. Final proof reading was made by both authors.
Conflict of interest:
None declared.
Financial support and sponsorship:
Nil.
REFERENCES
- 1.Masic I. Scientometric Analysis: a Technical Need for Medical Science Researchers Either as Authors or as Peer Reviewers. J. Res Pharm Pract. 2016;5:1–6. doi: 10.4103/2279-042X.176562. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Masic I, Begic E. Meta-analysis as Statistical and Analytical Method of Journal’ Content Scientific Evaluation. Acta Inform Med. 2015 Feb;23(1):4–11. doi: 10.5455/aim.2015.23.4-11. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Checketts JX, Cook C, Imani S, Duckett L, Vassar M. An Evaluation of Reporting Guidelines and Clinical Trial Registry Requirements Among Plastic Surgery Journals. Ann Plast Surg. 2018;81(2):215–219. doi: 10.1097/SAP.0000000000001476. [DOI] [PubMed] [Google Scholar]
- 4.PubMed [Internet] PubMed. [2020 Jul 29]. Available from: https://pubmed.ncbi.nlm.nih.gov/
- 5.Company W of SG a Clarivate. Web of Science Master Journal List [Internet]. Web of Science Group, a Clarivate company. [2020 Jul 29]. Available from: https://mjl.clarivate.com/
- 6.Jankovic SM, Masic I. Methodological Errors in Clinical Studies Published by Medical Journals of Ex-Yugoslav Countries. Acta Inform Med. 2020 Jun;28(2):84–91. doi: 10.5455/aim.2020.28.84-91. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Vesterinen HM, Vesterinen HV, Egan K, Deister A, Schlattmann P, Macleod MR, et al. Systematic survey of the design, statistical analysis, and reporting of studies published in the 2008 volume of the Journal of Cerebral Blood Flow and Metabolism. J Cereb Blood Flow Metab Off J Int Soc Cereb Blood Flow Metab. 2011 Apr;31(4):1064–1072. doi: 10.1038/jcbfm.2010.217. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Janković S. MEDRAT. Kragujevac, Serbia: 2016. Dizajn istraživanja. [Google Scholar]
- 9.Masic I, Jankovic SM. Meta-Analysing Methodological Quality of Published Research: Importance and Effectiveness. Stud Health Technol Inform. 2020 Jun 26;272:229–232. doi: 10.3233/SHTI200536. [DOI] [PubMed] [Google Scholar]
- 10.Vilibić M, Novački K, Vrkić N, Karlović D. Low BDNF concentrations in schizophrenia may be caused by non-linearity of BDNF correlation with age caused by its interaction with sex. Alcohol Psychiatry Res J Psychiatr Res Addict. 2018 Jun 15;54(1):21–32. [Google Scholar]
- 11.Blažinović I, Orlović I, Karlović D, Peitl V. Comparison of Clinical and Sociodemographic Characteristics of Patients with Schizophrenia Treated Stationary and at Day Hospital. Arch Psychiatry Res Int J Psychiatry Relat Sci. 2019 Dec 16;55(2):127–138. [Google Scholar]
- 12.Jiang B, Ding C, Yao G, Yao C, Zhang Y, Ge J, et al. Iintervention Effect of Folic Acid and Vitamin B12 on Vascular Cognitive Impairment Complicated With Hyperhomocysteinemia. J Med Biochem. 2013 Apr 1;33(2):169–174. [Google Scholar]
- 13.Zogović D, Pešić V, Dmitrašinović G, Dajak M, Plećaš B, Batinić B, et al. Pituitary-Gonadal, Pituitary-Adrenocortical Hormones and IL-6 Levels Following Long-Term Magnesium Supplementation in Male Students. J Med Biochem. 2014 Jul 1;33(3):291–298. [Google Scholar]
- 14.Sivaci R, Demir S, Koken T, Sivaci Y, Yilmaz S. Biochemical Effects of Low-Flow Anesthesia with Inhalation agents in Patients undergoing Laparoscopic Surgery. J Med Biochem. 2012 Jan 1;31(1):53–59. [Google Scholar]
- 15.de Almeida O, de Oliveira Sampaio A, Conceição M, Paranhos V, Mello D, Vale R, et al. Muscular Strength and Hydroxyproline Concentration in Urine After Different Flexibility Training Protocols. J Med Biochem. 2010 Apr 1;29(2):89–94. [Google Scholar]
- 16.Al-Zeidaneen S, Ahmad M, Al-Ebous A. Evaluation of the influence of treatment exposure on neuropeptide leptin hormone and obesity risk among Jordanian breast cancer women. Acta Fac Medicae Naissensis. 2019;36(2):150–158. [Google Scholar]
- 17.Stefanović A, Ristovski-Kornic D, Kotur-Stevuljević J, Spasojević-Kalimanovska V, Vekić J, Miljković M, et al. Alterations of HDL Particles in Children with End-Stage Renal Disease. J Med Biochem. 2017 Aug 28;36(4):358–365. doi: 10.1515/jomb-2017-0019. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Đorđević N, Stanojlović M, Milenković S, Stojiljković P, Kocić M, Golubović I. Perioperative and Early Postoperative Outcome of Proximal Femoral Nailing for Stable and Unstable Trochanteric Fractures. Acta Fac Medicae Naissensis. 2016 Mar 1;33(1):43–47. [Google Scholar]
- 19.Živković VD, Stanković I, Dimitrijević L, Čolović H, Spalević M, Savić N. The Role of Early Habilitation in Infants with Congenital Brachial Palsy. Acta Fac Medicae Naissensis. 2018 Dec 1;35(4):289–298. [Google Scholar]
- 20.Kozomara S, Krstić I. Effects of Dynamic Exercise and Dynamic Resistance Training in Rehabilitation of Patients after Surgical Coronary Revascularization. Acta Fac Medicae Naissensis. 2017 Jun 1;34(2):169–178. [Google Scholar]
- 21.Nešić I, Savić V, Kolarević A. Investigation of efficacy of anti-aging liposomal intimate gel: An in vivo long-term study. Acta Fac Medicae Naissensis. 2020;37(1):48–56. [Google Scholar]
- 22.Gašpar M, Glavina A, Grubišić K, Sabol I, Bušić M, Mravak-Stipetić M. The Oral Cavity State in Renal Transplant Recipients. Acta Stomatol Croat. 2015 Sep 22;49(3):204–213. doi: 10.15644/asc49/3/2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Haliti F, Jurić H. Povezanost dentalne traume, anksioznosti i agresivnog ponašanja kod djece dobi između 7 i 14 godina na Kosovu. Acta Stomatol Croat. 2017;51(1):3–12. doi: 10.15644/asc51/1/1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Alajbeg IŽ, Borić Brakus R, Brakus I. Comparison of Amitriptyline with Stabilization Splint and Placebo in Chronic TMD Patients: a Pilot Study. Acta Stomatol Croat. 2018 Jun 12;52(2):114–122. doi: 10.15644/asc52/2/4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Čelebić A, Peršić S, Kovačić I, Buković D, Lešić N, Rener-Sitar K. Comparison of Three Prosthodontic Treatment Modalities for Patients with Periodontally Compromised Anterior Mandibular Teeth: A 2-year follow-up study. Acta Stomatol Croat. 2019 Mar 19;53(1):4–16. doi: 10.15644/asc53/1/1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Muhić E, Plančak D, Lajnert V, Muhić A. Predictors of Job Satisfaction in Dental Professionals of the Bosnia and Herzegovina Federation. Acta Stomatol Croat. 2016 Sep 23;50(3):222–229. doi: 10.15644/asc50/3/4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Erceg T, Buljan Flander G, Brezinšćak T. The Relationship Between Compulsive Internet Use and Symptoms of Depression and Anxiety in Adolescence. Alcohol Psychiatry Res J Psychiatr Res Addict. 2018 Dec 20;54(2):101–112. [Google Scholar]
- 28.Grizelj M, Crnković D, Kostanjšak L, Vrkić N, Karlović D. Comparison of Lithium Concentration in Serum, Plasma and Erythrocytes. Alcohol Psychiatry Res J Psychiatr Res Addict. 2017 Dec 15;53(2):99–114. [Google Scholar]
- 29.Lotar Rihtarić M, Vrselja I, JandrićA Nišević A. Relationship between Alcohol Consumption and Violent Offending: Personality as a Contributing Factor. Alcohol Psychiatry Res J Psychiatr Res Addict. 2016 Dec 16;52(2):105–114. [Google Scholar]
- 30.Sidlauskiene A, Strukcinskiene B, Raistenskis J, Stukas R, Strukcinskaite V, Buckus R. The association between the level of physical activity with spinal posture and physical fitness parameters in early adolescence. Vojnosanit Pregl. 2019;76(12):1209–1216. [Google Scholar]
- 31.Grbović V, Stefanović S, Đukić S, Nurković J, Zdravković-Petrović N, Parezanović-Ilić K, et al. The effects of the physical procedures in patients with diabetic neuropathy. Vojnosanit Pregl. 2019;76(8):787–794. [Google Scholar]
- 32.Stefanović Z, Donfrid B, Jovanović T, Zorić Z, Radojević-Popović R, ZoranovićA U. Hyperbaric oxygenation in prevention of amputations of diabetic foot. Vojnosanit Pregl. 2020;77(4):363–372. [Google Scholar]
- 33.Ivanović S, Trgovčević S. Risk factors for fear of falling in the elderly in Serbia. Vojnosanit Pregl. 2018;75(8):764–772. [Google Scholar]
- 34.Gajić M, Lalić M, Kalevski K, Marjanović M. Oral health-related quality of life among Belgrade adolescents. Vojnosanit Pregl. 2018;75(1):8–15. [Google Scholar]
- 35.Kim JY, Moon S, Park JH, Cho HJ, Song JH, Jeon W, et al. Effect of transported hospital resources on neurologic outcome after out-of-hospital cardiac arrest. Signa Vitae J Intesive Care Emerg Med. 2019 May 1;15(1):51–58. [Google Scholar]
- 36.Karlović Z, Vladić D, Perić M, Mihalj M, Zadro Ž, Majerić-Kogler V. The impact of early percutaneous tracheotomy on reduction of the incidence of ventilator associated pneumonia and the course and outcome of ICU patients. 2018 Mar 27;14(1):75–80. [Google Scholar]
- 37.Prkaćin I, Adam VN, CavrićA G, Svaguša T, Kovačić M, Kovaćević I. Direct oral anticoagulant therapy and drug interactions in patients with atrial fibrillation. 2017 May 8;13(S1):68–70. [Google Scholar]
- 38.Kosmač N, Vidmar G, Virag M, Osredkar J, Knežević I. Comparison of sevoflurane and propofol anaesthetic regimes in respect to the release of troponin I and cystatin C in off-pump myocardial revascularisation: a randomised controlled trial. 2016 Oct 10;12(S1):36–42. [Google Scholar]
- 39.Marić N, Mačković M, Udiljak N, Bekić D. An overview of thrombolytic therapy for pulmonary embolism: a single centre experience. Signa Vitae J Intesive Care Emerg Med. 2015 Jun 1;10(Suppl. 1):68–71. [Google Scholar]
- 40.Marciuš VŽ. Vrednotenje zadovoljstva bolnikov z radioterapevtskimi storitvami na Onkološkem inštitutu Ljubljana. Onkol Med-Sci J. 2019 May 31;23(1):10–17. [Google Scholar]
- 41.Kukovica MG, Hribernik N, Herzog A, Čufer T. Febrilna nevtropenija in nevtropenija višje stopnje ob adjuvantnem zdravljenju nedrobnoceličnega raka pljuč v vsakodnevni praksi. Onkol Med-Sci J. 2018 Jul 7;22(1):60–66. [Google Scholar]
- 42.Škof E, Mangaroski D. Naše izkušnje z zdravilom bevacizumab v primarnem zdravljenju epitelijskega raka jajčnikov. Onkol Med-Sci J. 2020 May 12;24(1):20–24. [Google Scholar]
- 43.Demir S, Müslümanoğlu MH, Müslümanoğlu M, Başaran S, Çalay ZZ, Aydıner A, et al. TWIST1 Gene Expression as a Biomarker for Predicting Primary Doxorubicin Resistance in Breast Cancer. Balk J Med Genet BJMG. 2019 Dec;22(2):25–30. doi: 10.2478/bjmg-2019-0025. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Terzic M, Jakimovska M, Fustik S, Jakovska T, Sukarova-Stefanovska E, Plaseska-Karanfilska D. Cystic Fibrosis Mutation Spectrum in North Macedonia: A Step Toward Personalized Therapy. Balk J Med Genet BJMG. 2019 Aug 28;22(1):35–40. doi: 10.2478/bjmg-2019-0009. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Vachev T, Stoyanova V, Ivanov H, Minkov I, Popov N. Investigation of fasciculation and elongation protein ζ-1 (FEZ1) in peripheral blood reveals differences in gene expression in patients with schizophrenia. Balk J Med Genet BJMG. 2015 Dec 30;18(1):31–38. doi: 10.1515/bjmg-2015-0003. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Khabour O, Amarneh B, Bani Hani E, Lataifeh I. Associations Between Variations in TPH1, TPH2 and SLC6A4 Genes and Postpartum Depression: A Study in the Jordanian Population. Balk J Med Genet BJMG. 2013;16(1):41–48. doi: 10.2478/bjmg-2013-0016. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Sobti R, Sharma V, Abitew A, Berhane N, Mahdi S, Askari M, et al. IL-18 Gene Promoter Region 607C/A Polymorphism in HIV-1 Infected North Indian Population. Balk J Med Genet BJMG. 2011 Dec;14(2):41–48. doi: 10.2478/v10034-011-0046-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Wertheimer V, Antekolovic L, Matkovic BR. Muscle Damage Indicators after Land and Aquatic Plyometric Training Programmes. Montenegrin J Sports Sci Med. 2018;7(1):13–19. [Google Scholar]
- 49.Arazi H, Asadi A, Chegini J. Perceived Muscle Soreness, Functional Performance and Cardiovascular Responses to an Acute Bout of Two Plyometric Exercises. Montenegrin J Sports Sci Med. 2016;5(2):17–23. [Google Scholar]
- 50.Muratovic A, Vujovic D, Hadzic R. Comparative Study of Anthropometric Measurement and Body Composition between Elite Handball and Basketball Players. Montenegrin J Sports Sci Med. 2014;3(2):19–22. [Google Scholar]
- 51.Zhang Y, Nepocatych S, Katica CP, Collins AB, Casaru C, Balilionis G, et al. Effect of Half Time Cooling on Thermoregulatory Responses and Soccer-Specific Performance Tests. Montenegrin J Sports Sci Med. 2014;3(1):17–22. [Google Scholar]
- 52.Brás R, Esteves D, Rodrigues RG, Duarte P, Gouveia A, O’Hara K, et al. Evaluation of Risks and Benefits of Physical Activity of Hypertensives and Normotensives: Fighting a Societal Burden. Montenegrin J Sports Sci Med. 2020;9(1):11–18. [Google Scholar]
- 53.Duc NM, Huy HQ, Keserci B, Thong PM. Gender Disparity in Vietnamese Radiological Societies: a Preliminary Observational Study. Acta Inform Med. 2020 Mar;28(1):71–74. doi: 10.5455/aim.2020.28.71-74. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 54.Askari-Majdabadi H, Valinejadi A, Mohammadpour A, Bouraghi H, Abbasy Z, Alaei S. Use of Health Information Technology in Patients Care Management: a Mixed Methods Study in Iran. Acta Inform Med. 2019 Dec;27(5):311–317. doi: 10.5455/aim.2019.27.311-317. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55.Selmanovic S, Beganlic A, Salihefendic N, Ljuca F, Softic A, Smajic E. Therapeutic Effects of Curcumin on Ultrasonic Morphological Characteristics of Liver in Patients with Metabolic Syndrome. Acta Inform Med. 2017 Sep;25(3):169–174. doi: 10.5455/aim.2017.25.165-174. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 56.Gholipour Baradari A, Alipour A, Firouzian A, Moarab L, Emami Zeydi A. A Double-Blind Randomized Clinical Trial Comparing the Effect of Neostigmine and Metoclopramide on Gastric Residual Volume of Mechanically Ventilated ICU Patients. Acta Inform Med. 2016;24(6):385–389. doi: 10.5455/aim.2016.24.385-389. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 57.Lajqi N, Ilazi A, Kastrati B, Islami H. Comparison of Glucocorticoid (Budesonide) and Antileukotriene (Montelukast) Effect in Patients with Bronchial Asthma Determined with Body Plethysmography. Acta Inform Med. 2015 Dec;23(6):347–351. doi: 10.5455/aim.2015.23.347-351. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 58.Berisha M, Hyseni N, Statovci S, Grajqevci S, Xhiha B. Endoscopic management of vesicoureteral reflux in children in kosova. Acta Inform Med. 2014 Jun;22(3):167–169. doi: 10.5455/aim.2014.22.167-169. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 59.Gojkov-Vukelic M, Hadzic S, Dedic A, Konjhodzic R, Beslagic E. Application of a diode laser in the reduction of targeted periodontal pathogens. Acta Inform Med. 2013 Dec;21(4):237–240. doi: 10.5455/aim.2013.21.237-240. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 60.Forootan M, Tabatabaeifar M, Fatemi M, Darvishi M. Efficacy of Chemoradiotherapy on Health-Related Quality of Life in Esophageal Cancer Patients with Dysphagia. Mater Sociomed. 2019 Dec;31(4):246–252. doi: 10.5455/msm.2019.31.246-252. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 61.Latsou D, Geitona M. The Effects of Unemployment and Economic Distress on Depression Symptoms. Mater Sociomed. 2018 Oct;30(3):180–4. doi: 10.5455/msm.2018.30.180-184. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 62.Moghadam MP, Nourisancho H, Shahdadi H, Shahraki S, Azarkish B, Balouchi A. Effects of Home Care Training on the Self Efficasy of Patients with Beta Thalassemia Major. Mater Sociomed. 2016 Oct;28(5):357–360. doi: 10.5455/msm.2016.28.357-360. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 63.Jafarzadeh L, Motamedi A, Behradmanesh M, Hashemi R. A Comparison of Serum Levels of 25-Hydroxy Vitamin D in Pregnant Women ar Risk for Gestational Diabetes Mellitus and Women Without Risk Factors. Mater Sociomed. 2015 Oct;27(5):318–322. doi: 10.5455/msm.2015.27.318-322. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 64.Jafari F, Mobasheri M, Mirzaeian R. Effect of diet education on blood pressure changes and interdialytic weight in hemodialysis patients admitted in hajar hospital in shahrekord. Mater Sociomed. 2014 Oct;26(5):318–320. doi: 10.5455/msm.2014.26.318-320. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 65.Petrovic M, Kesic L, Obradovic R, Savic Z, Mihailovic D, Obradovic I, et al. Comparative analysis of smoking influence on periodontal tissue in subjects with periodontal disease. Mater Sociomed. 2013;25(3):196–198. doi: 10.5455/msm.2013.25.196-198. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 66.Garousi S, Garrusi B, Divsalar F, Divsalar K. Body satisfaction and management in Iranian students. Mater Sociomed. 2012;24(1):34–37. doi: 10.5455/msm.2012.24.34-37. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 67.Rohani F, Asadi R, Mirboluk AA, Soheilipour F. Letrozole Effect on Final Height of Patients with Constitutional Delay of Growth and Puberty. Med Arch. 2019 Oct;73(5):307–310. doi: 10.5455/medarh.2019.73.307-310. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 68.Hoxha A, Shehu S, Deveja R, Qirjazi T. Impact of Clopidogrel Loading for Coronarography on Bleeding After Urgent First Time CABG. Med Arch. 2018 Nov;72(5):319–324. doi: 10.5455/medarh.2018.72.319-324. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 69.Torkaman A, Yazdi H, Hosseini MG. The Results of Single Bundle Versus Double Bundle ACL Reconstruction Surgery, a Retrospective Study and Review of Literature. Med Arch. 2016 Oct;70(5):351–353. doi: 10.5455/medarh.2016.70.351-353. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 70.Vanis-Vatrenjak S, Mesic A, Abdagic I, Mujezinovic D, Zvizdic Z. Quality and Safety of General Anesthesia with Propofol and Sevoflurane in Children Aged 1-14 Based on Laboratory Parameters. Med Arch. 2015 Aug;69(4):218–221. doi: 10.5455/medarh.2015.69.218-221. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 71.Haxhibeqiri-Karabdic I, Hasanovic A, Kabil E, Straus S. Improvement of ejection fraction after coronary artery bypass grafting surgery in patients with impaired left ventricular function. Med Arch. 2014 Oct;68(5):332–334. doi: 10.5455/medarh.2014.68.332-334. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 72.Djedovic M, Rustempasic N, Djedovic S, Totic D, Solakovic S, Mujanovic E, et al. Comparison of transperitoneal and retroperitoneal approach in aorto iliac occlusive disease. Med Arch. 2013;67(4):249–251. doi: 10.5455/medarh.2013.67.249-251. [DOI] [PubMed] [Google Scholar]
- 73.Rahmanovic E, Rifatbegovic Z. Recovery of liver function after surgical procedure of penetrative and nonpenetrative liver injury. Med Arch. 2012;66(3):166–168. doi: 10.5455/medarh.2012.66.166-168. [DOI] [PubMed] [Google Scholar]
- 74.Zonić-Imamović M, Imamović S, Čičkušić A, Delalić A, Hodžić R, Imamović M. Effects of Treating an Overactive Urinary Bladder in Patients with Multiple Sclerosis. Acta Medica Acad. 2019 Dec;48(3):271–277. doi: 10.5644/ama2006-124.267. [DOI] [PubMed] [Google Scholar]
- 75.Bajrić E, Kobašlija S, Jurić H, Huseinbegović A, Zukanović A. The Reliability and Validity of the Three Modified Versions of the Children’s Fear Survey Schedule-Dental Subscale of 9-12 Year Old Children in a Clinical Setting in Bosnia and Herzegovina. Acta Medica Acad. 2018 May;47(1):1–10. doi: 10.5644/ama2006-124.209. [DOI] [PubMed] [Google Scholar]
- 76.Račić M, Joksimović BN, Cicmil S, Kusmuk S, Ivković N, Hadživuković N, et al. The effects of interprofessional diabetes education on the knowledge of medical, dentistry and nursing students. Acta Medica Acad. 2017 Nov;46(2):145–154. doi: 10.5644/ama2006-124.199. [DOI] [PubMed] [Google Scholar]
- 77.Godinjak A, Iglica A, Rama A, Tančica I, Jusufović S, Ajanović A, et al. Predictive value of SAPS II and APACHE II scoring systems for patient outcome in a medical intensive care unit. Acta Medica Acad. 2016 Nov;45(2):97–103. doi: 10.5644/ama2006-124.165. [DOI] [PubMed] [Google Scholar]
- 78.Meneses Calderón J, González Sánchez I, Aburto Huacuz G, Alonso Barreto AS, Del Carmen Colín Ferreyra M, Mendieta Zerón H. Trends of inflammatory markers and cytokines after one month of phototherapy in patients with rheumatoid arthritis. Acta Medica Acad. 2015;44(2):102–108. doi: 10.5644/ama2006-124.137. [DOI] [PubMed] [Google Scholar]
- 79.Krstović-Spremo V, Račić M, Joksimović BN, Joksimović VR. The effects of diabetes mellitus and hypertension on work productivity. Acta Medica Acad. 2014;43(2):122–133. doi: 10.5644/ama2006-124.111. [DOI] [PubMed] [Google Scholar]
- 80.Loghmani A, Golshiri P, Zamani A, Kheirmand M, Jafari N. Musculoskeletal symptoms and job satisfaction among office-workers: a cross-sectional study from Iran. Acta Medica Acad. 2013;42(1):46–54. doi: 10.5644/ama2006-124.70. [DOI] [PubMed] [Google Scholar]
- 81.Iglica A, Godinjak A, Begić E, Hodžić E, Zvizdić F, Kukavica N, et al. Therapeutic hypothermia as a treatment option after out-ofhospital cardiac arrest: our experience. Med Glas (Zenica) 2019 Aug 1;16(2) doi: 10.17392/1040-19. [DOI] [PubMed] [Google Scholar]
- 82.Rakanović Todić M, Maleškić S, Burnazović Ristić L, Kulo Ćesić A, Kusturica J, Prnjavorac B. Role of inhaled corticosteroids in the exacerbation rate of moderate chronic obstructive pulmonary disease. Med Glas (Zenica) 2018 Feb 1;15(1):23–28. doi: 10.17392/935-18. [DOI] [PubMed] [Google Scholar]
- 83.Vegar Zubović S, Kristić S, Sefić Pašić I. Relationship between ultrasonographically determined kidney volume and progression of chronic kidney disease. Med Glas (Zenica) 2016 Aug 1;13(2):90–94. doi: 10.17392/852-16. [DOI] [PubMed] [Google Scholar]
- 84.Okanović A, Prnjavorac B, Jusufović E, Sejdinović R. Alpha-lipoic acid reduces body weight and regulates triglycerides in obese patients with diabetes mellitus. Med Glas (Zenica) 2015 Aug;12(2):122–127. doi: 10.17392/798-15. [DOI] [PubMed] [Google Scholar]
- 85.Mujarić E, Prnjavorac B, Kalajdžija-Cero M, Šestić H, Rošić A, Sejdinović R, et al. Changes of natriuretic peptides concentration in early phase of acute myocardial infarction. Med Glas (Zenica) 2014 Feb;11(1):7–12. [PubMed] [Google Scholar]
- 86.Mehmedović A, Mesihović R, Prnjavorac B, Vanis N, Vukobrat-Bijedić Z, Borovac N, et al. Non-invasive liver fibrosis markers: use of serum levels of cytokines IL 1α and TGF ß1 in management of chronic liver diseases. Med Glas (Zenica) 2013 Feb;10(1):20–27. [PubMed] [Google Scholar]
- 87.Gholipour Baradari A, Emami Zeydi A, Espahbodi F, Aarabi M. The effect of intravenous vitamin C on the phosphorus level reduction in hemodialysis patients: a double blind randomized clinical trial. Med Glas (Zenica) 2012 Feb;9(1):37–41. [PubMed] [Google Scholar]
- 88.Stevanovic S, Nikolic M, Stankovic A. Dietary magnesium intake and coronary heart disease risk: a study from Serbia. Med Glas (Zenica) 2011 Aug;8(2):203–208. [PubMed] [Google Scholar]
- 89.Nouraei SM, Baradari AG, Jazayeri A. Does Remote Ischaemic Preconditioning Protect Kidney and Cardiomyocytes After Coronary Revascularization? A Double Blind Controlled Clinical Trial. Med Arch. 2016 Oct;70(5):373–378. doi: 10.5455/medarh.2016.70.373-378. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 90.Ðug H, Jagodić S, Ahmetović-Ðug J, Selimović Z, Sulejmanović A. Predicting recurrence of non-muscle-invasive bladder cancer after transurethral resection. Med Glas (Zenica) 2016 Feb 1;13(1):56–61. doi: 10.17392/824-16. [DOI] [PubMed] [Google Scholar]
- 91.Asheghan M, Aghda AK, Hashemi E, Hollisaz M. Investigation of the Effectiveness of Acupuncture in the Treatment of Frozen Shoulder. Mater Sociomed. 2016 Jul 24;28(4):253–257. doi: 10.5455/msm.2016.28.253-257. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 92.Stanetić K, Petrović V, Marković B, Stanetić B. The Presence of Stress, Burnout Syndrome and the Most Important Causes of Working Stress Among Physicians in Primary Health Care–an Observational Study from Banja Luka, Bosnia and Herzegovina. Acta Medica Acad. 2019 Aug;48(2):159–166. doi: 10.5644/ama2006-124.254. [DOI] [PubMed] [Google Scholar]
- 93.Alimian M, Mohseni M, Safaeian R, Faiz SHR, Majedi MA. Comparison of hydroxyethyl starch 6% and crystalloids for preloading in elective caesarean section under spinal anesthesia. Med Arch. 2014 Aug;68(4):279–281. doi: 10.5455/medarh.68.279-281. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 94.Alkhaldi HM, Salaita WM, Shabaneh MA, Al-Horut MI, Aldabbas RMAA, Uraiqat AA. Postoperative Outcome Comparison Between Pudendal Nerve Block and Caudal Block After Lateral Open Internal Sphincterotomy. Med Arch. 2015 Jun;69(3):187–189. doi: 10.5455/medarh.2015.69.187-189. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 95.Esparza M, Wild JR, Minnock C, Mohty KM, Truchan LM, Taljanovic MS. Ultrasound Evaluation of Radial Nerve Palsy Associated with Humeral Shaft Fractures to Guide Operative Versus Non-Operative Treatment. Acta Medica Acad. 2019 Aug;48(2):183–192. doi: 10.5644/ama2006-124.257. [DOI] [PubMed] [Google Scholar]
- 96.Arkin Funda Secik, Aras Gulfidan, Dogu Elif. Comparison of Artificial Neural Networks and Logistic Regression for 30-days Survival Prediction of Cancer Patients. Acta Inform Med. 2020;28(2):108–113. doi: 10.5455/aim.2020.28.108-113. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 97.Masic I, Kujundzic E. Avicena. Sarajevo: 2013. Science Editing in Biomedicine and Humanities; pp. 69–94. [Google Scholar]
- 98.Aljumili R. To Mr Jeffrey Beall: Bring forth your evidence, if you’re telling the truth! [June 15, 2020]. www.Refat.A.70@outlook.com.
- 99. [June 15, 2020]. http://citesandinsights.info/civ14i4.pdf.
- 100.Jankovic SM, Kapo B, Sukalo A, Masic I. Evaluation of Published Preclinical Experimental Studies in Medicine: Methodology Issues. Med Arch. 2019;73(5):298–302. doi: 10.5455/medarh.2019.73-298-302. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 101.Jankovic SM, Masic I. Evaluation of Preclinical and Clinical Studies Published in Medical Journals of Bosnia and Herzegovina: Methodology Issues. Acta Inform Med. 2020;28(1):4–11. doi: 10.5455/aim.2020.28.4-11. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 102.Masic I. Medical Publications and Scientometrics. Journal of Research in Medical Sciences. 2013 Jun;18(6):624–630. [PMC free article] [PubMed] [Google Scholar]
- 103.Masic I, Jakovljevic M, Sinanovic O, Gajevic S, Spiroski M, Jusufovic R, et al. The Second Mediterranean Seminar on Science Writing, Editing and Pubishing (SWEP 2018), Sarajevo, December 8th, 2018. Acta Inform Med. 2018 Dec;26(4):284–296. doi: 10.5455/aim.2018.26.284-296. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 104.Masic I, Begic E, Donev MD, Gajovic S, Gasparyan YA, Jakovljevic M, Milosevic BD, Sinanovic O, Sokolovic S, Uzunovic S, Zerem E. Sarajevo Declaration on Integrity and Visibility of Scholarly Journals. Croat Med J. 2016;57:527–529. doi: 10.3325/cmj.2016.57.527.529. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 105.Masic I, Donev D, Sinanovic O, Jakovljevic M, Zerem E, Mi-losevic BD, Gajovic S, Gasparyan AY, Sokolovic S, Salkic N, Uzunovic S, Dobric S, Kurti F, Begic E, Kurtcehajic A. The First Mediterranean Seminar on Science Writing, Editing and Publishing, Sarajevo, December 2-3, 2016. Acta Inform Med. 2016 Dec;24(6):424–435. doi: 10.5455/aim.2016.24.424-435. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 106. [June 15, 2020]. https://www.scopus.com/sourceid/21100899502.
- 107. [June 15, 2020]. www.foliamedica.mf.unsa.ba/index.php/FM.
- 108.Shoja Mohammadali M., Arynchyna Anastasia., editors. London: Willey Blackwell; 2019. A Guide to the Scientific Career; pp. 163–178. [Google Scholar]
- 109.Masic I, Hodzic A, Mulic S. Ethics in Medical Research and Publication. International Journal of Preventive Medicine. 2014;5(9):1073–1082. [PMC free article] [PubMed] [Google Scholar]