Abstract
Background:
Knowledge production and evaluation are two important functions of health research system (HRS). In this article, we aimed to reveal the correlation between evaluation of health research organizations and health knowledge production promotion.
Methods:
A comprehensive evaluation system was developed to evaluate the academic performance of national medical science universities on an annual basis. It assess following domains; stewardship, capacity building and knowledge production. Measurable indicators for each domain were assigned, a ‘research profile’ for each department was provided. In this study, we compared the results of annually national Health Research System evaluation findings during 2005–2008.
Results:
The number of scientific articles has been increased from 4672 to 8816 during 2005 to 2008. It is mentionable that, the number of articles which has been published in indexed data bases has risen too. This fact could be related to directed policy for more international publication of scientific articles from Iran. The proportion of total articles to the number of academic members was 1.14 in 2008, comparing to 0.84 in 2005. It means that this proportion have increased about twice (0.7 Vs 0.45) during mentioned time. Moreover, other scientific products such as authored books based on domestic researches and cited articles in textbooks have increased according to special attention to knowledge production by policy makers.
Conclusion:
We conclude that Health System Research evaluation could be used as a mean for implementing policies and promoting knowledge production.
Keywords: Health, Research, Evaluation, Knowledge, Production, Iran
Introduction
Knowledge management is defined by World Health Organization (WHO) as, “a set of principles, tools and practices that enable people to create knowledge, and to share, translate and apply what they know to create value and improve effectiveness” (1).In recent decades, knowledge managers have focused on developing and enriching knowledge production strategies (2). In this regard, Health Research System (HRS) has a prominent responsibility in scaling up knowledge production (3).
HRS has certain special functions that help filling the gap between Research and Development (R&D), especially in developing countries (2, 4). One of these main functions is Monitoring and Evaluation (5). This function of the HRS is an operational component of stewardship and is useful as a tool for promoting the other function of HRS such as creating and applying research. HRS evaluation could be used for expanding policies, emerging trends in health research, knowledge generation, and technology development and making recommendations for future national policies (6).
Following the stated mission of the World Health Organization (WHO) as, “generation of appropriate knowledge and its utilization towards improved health”, the Department of Research and Technology of the MOHME in the Islamic Republic of Iran was tasked with the responsibility for designing and implementing the HRS functions and policies especially monitoring and evaluation (7). The main focus of this evaluation system was knowledge production to promote the academic performance of the researchers with an emphasis on promoting international scientific publications. Evaluation of research activities in terms of knowledge production was performed based on stakeholders agreed criteria.
The present paper aims to assess the contribution of the HRS evaluation on promotion of HRS performance and health knowledge production.
Materials and Methods
During last decade, the Department of Research and Technology of the Ministry of Health and Medical Education (MOHME) of I.R. Iran (as the main policy making organization in biomedical and health related research) concentrated on knowledge management promotion. Scientific activity is traditionally measured by the production of scientific publications, i.e. the number of articles published in scientific journals. Besides one decisive criterion in the realization of research policy is the number or volume of valid scientific research studies conducted by specialists in different fields (4).
A comprehensive evaluation system was developed to evaluate the academic performance of national medical science universities on an annual basis. Through a collaborative effort the key HRS authorities developed certain measures. A pilot study was conducted to establish the applicability of the evaluation measures adopted. In each year, the evaluation procedures were reviewed and renewed. Based on certain policies some measures were changed in line with stakeholders’ views and the objective results obtained. “Stewardship”, “Capacity Building”, and “Knowledge Production” were three domains that considered in evaluation process. (Table 1) shows the Medical Sciences University Evaluation Form.
Table 1:
Medical Sciences University Evaluation Form
| Domain | Subject | Measure | Score |
|---|---|---|---|
| Strategic Planning | Existence of strategic program, report of annual program evaluation | 0–100 | |
| Stewardship | Priority Setting | Existence of research priorities, Percentage of research projects in line with research priorities | 0–100 |
| Ethics committee Activities | Percentage of referred proposals to ethic committee | 0–100 | |
| Capacity Building | International Congress | Held international congress confirmed bycabinet | Each case 100 |
| National congress | Held national congress confirmed by Health ministry | 0–100 | |
| Research related Workshops | Held required workshop based on Research Council confirmation | 0–150 | |
| Scientific Awards | Get award from Razi or Kharazmi festivals | First 100 Second 70 Third 50 Student award 50 |
|
| Book authoring | Books which at least 1% of their references are authors self citations | Each case 25 | |
| Citation to articles in textbooks | Citation to articles in credible textbooks | Each case 15–25 | |
| Innovations and patents | Practical innovations and patents reported in scientific journals | National cases: 2–10 International cases: 5–100 | |
| Paper presentation in national and international congresses | Paper presentation in oral or poster form, in national and international congresses and papers indexed in ISI proceedings | Presented paper in national congress: Each case 0.5 Presented paper in Internationalcongress
|
|
| Knowledge on Production | Indexed articles in ISI/Web of Science and PubMed/Medline | Indexed articles in PubMed/Medline and ISI /Web of Science by considering the impact factor with two coefficients as the quality criteria of a journal. | Each article indexed in ISI/WOS :25+2(Impact Factor) Each article indexed in PubMed/Medline: 25 |
| Indexed articles in other international indexing databases(such as Scopus and Chemical Abstract and etc.) | Indexed articles in other international indexing databases ; Scopus and Chemical Abstract, Biological Abstract, embase, Index copernicous, IMEMRO | Each case 10–15 | |
| Nonindexed published articles in national and international scientific journals | Non-indexed published articles in journals witch have accredited by the Iranian Ministry of Science, Research and Technology (MSRT) and the Ministry of Health and Medical Education (MOHME) and credible foreign journals | Each case 5 |
According to indexing type, article type and the place of publication, all documents were classified and cleaned in order to avoid double counting. The articles were divided into 5 types including of original research article, review article, brief report or short communication, case report and letter to the editor.
The evaluation was carried out by a team consisting of research experts at the office of undersecretary for research. All team members applied the developed measures to Health Research System in 40 universities of medical sciences annually. The measures were assessed during a meeting with the representative of each university. The evaluation forms were filled out based on controlling presented official documents.
Results
Figure 1 shows the results of the 2005–2008 annual HRS evaluation report. The data presents the total number of published articles and medical researchers during mentioned time period. The total number of published articles has increased from 4672 to 8816 during mentioned period. It is noticeable that during the same period total number of medical researchers has increased 1.2%. The results were analyzed according to cited articles in widely recognized data bases. Figure 2 provides detailed information about the growth in the total number of cited articles.
Fig. 1:
Annual academic and research paper publication in 2005–2008
Fig. 2:
Articles published during 2006–2008 period
The number of indexed articles in reliable and widely recognized data bases has risen by 127%. Although articles sited in ISI/PubMed have decreased in 2008, the number of published articles in Iranian journals indexed in ISI/PubMed has increased by 36% (from 654 papers in 2007 to 893 in 2008). Noticeably, the proportion of ISI Web of Science/PubMed indexed articles per medical researcher has increased from 0.09 to 0.33.
Retrieved data from 2005 to 2008 show that the trend of published articles cited in other recognized data bases including Scopus, EMBASE, Chemical Abstract, Index Medicus, EMRO and Index Copernicus was ascending. It is noticeable that the number of articles published in Iranian journals indexed in mentioned databases had increased by 14%. Also, the above Figure shows that the total number of non-indexed articles had a descending trend. A closer look at the data reveals that almost, 85% of articles were original, 7% case reports, 3% short communication papers, 3% letter to the editor and 2% review articles in 2008.
In 2008, national medical universities were ranked by the scores they received on different knowledge production axes. Table 2 shows the top ten ranking Iranian medical sciences universities in terms of their scores on knowledge production.
Table 2:
Knowledge production scores of top Ten Iranian Universities based on the 2008 evaluation results
| NO. | Medical Science University | 2005 | 2008 | ||||
|---|---|---|---|---|---|---|---|
|
| |||||||
| Academic Member | Total Articles | Knowledge Production Score | Academic Member | Total Articles | Knowledge Production Score | ||
| 1 | Tehran | 1236 | 1156 | 18178 | 1265 | 2346 | 43682 |
| 2 | Shahid Beheshty | 1049 | 656 | 9474 | 1115 | 1426 | 25711 |
| 3 | Isfahan | 637 | 382 | 5976 | 611 | 806 | 13778 |
| 4 | Shiraz | 538 | 299 | 5373 | 656 | 746 | 13494 |
| 5 | Tabriz | 553 | 269 | 3797 | 607 | 647 | 11269 |
| 6 | Mashhad | 584 | 320 | 4085 | 629 | 562 | 9422 |
| 7 | Iran | 721 | 245 | 3353 | 696 | 529 | 8043 |
| 8 | Baghiatallah | 183 | 185 | 2583 | 199 | 368 | 5979 |
| 9 | Ahvaz | 420 | 129 | 1678 | 472 | 317 | 5209 |
| 10 | Mazandaran | 209 | 187 | 2529 | 201 | 222 | 4063 |
The comparisons show that in terms of knowledge production, Tehran University of Medical Science stands in the first place. This university had 2346 scientific articles in 2008 that covered 26.6% of total national biomedical articles. Sitting in the second place is Shahid Beheshti University of Medical Science having published 1426 articles, which comprises 16% of total articles related to this university and Isfahan University of Medical Sciences University as the third rank had 806 scientific articles in 2008 that have increase about 110% growth in publication. It is mentionable that these top ten universities covered 72% of our country biomedical articles.
Findings of this study show that the proportion of total articles per research project was 0.84 in 2005 comparing with 1.14 in 2008. Moreover the proportion of total articles per medical researcher has increased about twice (0.7 Vs 0.45). In terms of knowledge production, the data collected in 2008 shows that 14,357 articles were presented in scientific colloquiums (9,938 in national congresses and 4,419 in international congresses) compared to the 6585 in 2005.
Furthermore, other scientific products such as published books, published scientific articles witch cited by credible textbooks have increased according to special attention to knowledge production by policy makers.
Discussion
The National Iranian Plan for Health Innovation and Science Development aims to increase the capacity to create and facilitate knowledge sharing and dissemination as two main policies (8). The Department of Research and Technology in the Ministry of Health and Medical Education has been developing systematic plans to realize these policy objectives in medical sciences universities. Furthermore, supporting national programs for knowledge management, promoting knowledge generation and establishing knowledge pattern are national level strategies recommended by WHO (2).
Health research has been broadly defined as “the generation of new knowledge using the scientific method to identify and deal with health problems” (9). Generated Knowledge will enable research stakeholders to improve implementation of Health research in order to promote health outcomes and health equity (3, 10).
Significant increase in number of scientific published papers has improved Iran’s scientific ranking word wide (11, 12). According to ESI (Essential Science Indicators), among a five-year period (2005–2009) Iran stands in the 19th rank for scientific output (13). This promotion could be as a result of the systematic approach to research management which enables research stakeholders to improve health outcomes and health equity (3, 10). Increased attention of health research policy makers to research projects in terms of quality as well as quantity, could contribute to raising the ratio of articles per projects (14).
According to the findings of this study the number of cited articles in ISI/PubMed has increased 1.3 times from 2005 to 2008. This fact could be as a result of revising research polices which offered more credit to the quality rather than sheer quantity. The number of Iranian biomedical journals has increased from 8 to 155 (1979 to 2010) (15). The number of Iranian medical journals that are indexed in scientific database increased from 53 in 2000 to 141 in 2008 (16). Increased accessibility to scientific data bases, advances in information and communication technology (ICT) and indexing Iranian journals in credible databases have promoted Iranian publication output (17).
The proportion of articles to academic member among, Tehran, Shahid Beheshti and Isfahan medical sciences universities were respectively 1.8 and 1.2 and 1.3.Ascending trend of this proportion among top universities leads to achievement of the comprehensive Medical Plan objectives.
In conclusion we believe that planned policies could be safely applied through HRS evaluation criteria. Therefore national policies may flow from the past, present and future by efficient use of available resources.
Ethical considerations
Ethical issues (Including plagiarism, Informed Consent, misconduct, data fabrication and/or falsification, double publication and/or submission, redundancy, etc) have been completely observed by the authors.
Acknowledgments
The authors wish to thank all the Research Experts of Deputy of Research and Technology and Medical science universities for assistance in data collection and review. This program was supported by the Ministry of Health and Medical Education /Deputy of Research & Technology, Islamic Republic of Iran. The authors declare that there is no conflict of interests.
References
- 1.Anonymous . Bridging the “Know-do” gap in global health :What is knowledge management. World Health Organization (WHO); 2010. Available from: http://www.who.int/kms/en. [Google Scholar]
- 2.Anonymous . Technical paper on Regional strategy for knowledge management to support public health. World Health Organization (WHO); 2006. Eastern Mediterranean Region Organization. Available from: http://www.who.int/kms/en. [Google Scholar]
- 3.Gonzalez-Block MA. Health policy and systems research agendas in developing countries. Health Res Policy Syst. 2004;2:6. doi: 10.1186/1478-4505-2-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Malekzadeh R, Mokri A, Azarmina P. Medical Science and Research in IRAN. Arch Irn Med. 2001;4(1):27–39. [Google Scholar]
- 5.Godin B, Gingras Y. The place of universities in the system of knowledge production. Research Policy. 2000;29:273–8. [Google Scholar]
- 6.Brutscher PB, Wooding S, Grant J. Health research evaluation frameworks: An international comparison. Canadian Academy of Health Sciences. 2008. Available from: http://www.rand.org/pubs/technical_reports/2008/RAND_TR629.sum.pdf.
- 7.Anonymous . Research Policy and Cooperation. World Health Organization (WHO); 2006. Eastern Mediterranean Region Organization. Available from: http://www.emro.who.int/rpc/. [Google Scholar]
- 8.Larijani B, Majdzadeh R, Delavari AR, Rajabi F, Khatibzadeh S, Esmailzadeh H, et al. Iran’s health innovation and science development plan by 2025. Iranian J Publ Health. 2009;38(suppl):13–16. [Google Scholar]
- 9.Anonymous . Health research-essential link to equity in development Commission on Health Research for Development. Oxford University Press; 1990. Available from: http://www.cohred.org/node/303. [Google Scholar]
- 10.Pang T, Sadana R, Hanney S, Bhutta ZA, Hyder AA, Simon J. Knowledge for better health — a conceptual framework and foundation for health research systems. Bull World Health Organ. 2003;81(11):815–20. [PMC free article] [PubMed] [Google Scholar]
- 11.Moin M, Mahmoudi M, Rezaei N. Scientific output of Iran at the threshold of the 21st century. Scientometrics. 2005;62(2):239–248. [Google Scholar]
- 12.Malekafzali H, Eftekhari MB, Peykari N, Gholami FS, Djalalinia Sh, Owlia P, et al. Research Assessment of Iranian Medical Universities, an Experience from a Developing Country. Iranian J Publ Health. 2009;38(Suppl):47–49. [Google Scholar]
- 13.Anonymous Ranking the top 20 countries of the world and the number of articles published in the databases ISI: Web of Science between the years 1999 until 2009. 2010. Ministry of Health and Medical Education, Deputy of Research & Technology, Scientometry Center. Available from: http://www.hbi.ir/files/res_sm/pages/docs/gozaresh/3.pdf.
- 14.Hanney S, Buxton M, Green C, Coulson D, Raftery J. An assessment of the impact of the NHS health technology assessment programme. Health technol assess. 2007;11(53):iii–iv. ix–xi, 1–180. doi: 10.3310/hta11530. [DOI] [PubMed] [Google Scholar]
- 15.Anonymous List of journals approved medical journal of the Commission 2010. 2010. Ministry of Health and Medical Education, Deputy of Research & Technology, The Commission for Accreditation & Improvement of Iranian Medical Journals. Available from: http://commission.hbi.ir/View/aproved.php.
- 16.Aminpour F, Kabiri P. Science production in Iran: the scenario of Iranian medical journals. JRMS. 2009;14(5):313–322. [PMC free article] [PubMed] [Google Scholar]
- 17.Ghaneirad MA, Tolou AAGH, Khosrou Khavar F. Factors, motives and challenges of knowledge production among scientific EITES. J Science and Technology Policy. 2008;1(2):71–85. [Google Scholar]