The Analysis of Health Research System Evaluation in Medical Sciences Universities

Abstract

Background:

Based on Iran by 2025 defined vision, we must to receive the first grade of science position in south western Asian region. Thus we need to have a comprehensive evaluation program.

Methods:

A comprehensive WHO Health Research System Analysis (HRSA)- based evaluation system was developed to evaluate the HRS in Iran. This article has explored the results of the five-year evaluation (2003–2008) and aims to introduce this method to other developing countries. Here we explore the results of research performance evaluation from 2002 to 2010 and by comparing the results with previous available information, we reveal the probable role of this method in research promotion and proposed approach to facilitate and expedite achieving the prospects for goals of health research based on the visions of Iran by 2025.

Results:

All of the indicators of stewardship and capacity building axes are received to their predefined levels. Moreover all of the medical science university research policies are based on their strategic plannings which are extracted from national visions of Iran by 2025. Most of the predefined goals in knowledge production domain had a significant grow trend but for more growth for commitments they should be closely follow.

Conclusion:

We developed an HRS-based comprehensive evaluation program to our national vision as well as our regional and international research competition.

Introduction

Evaluation as a general refers to the determination of the quality of a program based on the formulating a judgment (1, 2). In recent decades the evaluation of the universities and institutions has opened one of the most controversial and interesting area (3–5).

Different international universities and scientific institutes assessing and ranking methods have been developed in different countries based on specific aims and interests (6–9).

As each country has the special research area with specific vision and so many other considerable inclusive factors, validation of developed methods based on specific conditions and research strategies is the most preferable strategy for research promotion (6, 10, 11). In Iran, the Comprehensive Scientific Map of the Country has outlined a coordinated and dynamic collection for achieving the visionary goals of the country by 2025 (12–14). Considering above we developed an HRS-based (Health Research System-based) method of research components evaluation which helps health research policy makers to more evidence based approaches (15–18).

The present paper is intended to assess a part of comprehensive achievement of the intermediate and long term health research goals and define appropriate approach to facilitate and expedite achieving of goals of health research based on the visions of Iran by 2025.

Material and Methods

Aiming to access the flow, strengths and weaknesses of health research; in 2001 the Ministry of Health and Medical Education of Iran began a process to annual research performance evaluation of governmental medical sciences and their affiliated research institutions. Data collection forms were designed based on some indicators of WHO Health Research System Analysis (HRSA) considering the indicators of standardized international universities’ assessing and specific vision of health researches in our country by participation of key HRS policy makers and other health research stakeholders. Through the pilot study with participation of eight medical sciences universities, the validity and reliability of them were confirmed. It consists of 2 main parts; inputs including human resource and allocated research budget, and outputs covering HRS evaluation indicators. HRS indicators were scored in three axes based on effective HRS functions: Indicators of stewardship axes were the assessment of health research priority setting; five-year strategic plan formulation and monitoring; establishment and performance of Ethic Committee. In capacity building indicators consisted of short training programs and held the international and national congresses; reward achievements in scientific gatherings; Webometery of the university and their affiliated institutions; establishing and empowering student research group. Finally knowledge production indicators were number of articles indexed in ISI/Thomson and Pubmed/Medline; Number of articles indexed in other indexed databases; abstracts presented at national and international congresses; number and quality of research projects; published books; number of patents; applied interventional projects which improved health system, and citations of the published papers in textbooks and peer review journals. Each of these indexes had an individual score and final score was calculated by sum of scores.

In the present article, we explore the results of research performance evaluation from 2002 to 2010 and by comparing the results with previous available information, we reveal the probable role of this method in research promotion and proposed approach to facilitate and expedite achieving the prospects for goals of health research based on the visions of Iran by 2025.

Results

As the main research inputs; during the past decade there was a significant growing in research resources. The total number of academic members rose from 9610 to 12447. Whereas there was no any defined position for academic research members in early years of starting the program, in 2010, 289 professionals were registered as academic research members in classified academic categories.

The proportion of post graduate medical science students that are engaged in research to total medical science students had more than 60% growth.

On the other hand, the proportion of research budget per academic member has increased from 17.25 to 90.41 million RI (nearly fivefold). Now about 40% of health research budget is used for research project from which 70% is used for research project that are in the priority line.

In 2002, the ratio of approved projects per hundred researchers was estimated 43, whereas this ratio rose to 70 projects per one hundred researchers in 2010 that represents growth of 63%.

Under monitoring obtained achievements

We should focus on the obtained achievements that mainly reached to their predefined standards and now should be monitored and then on the goals and expected achievement that would be pursued. In the first category, the indicators of stewardship axes in their foundation and continues performance have reached to their predetermined standard (Table 1).

Table 1:

Under monitoring obtained achievements of Stewardship axes

Stewardship axes indicators		Predetermined standard out put
Health research priority setting	Foundation	Health research priority setting process is completed all of the medical sciences universities.
	Continues performance	In the medical sciences university’s entire at least %70 of research project approval and their budget allocation are based on the health research priorities.
Formulation of a five-year strategic plan		Designation of a five-year strategic plan and evaluation of objectives achievement
Ethics committee	Foundation	These committees are established in all of medical sciences university’s with appropriate composition of ethic committee members.
	Continues performance	Research projects have been referred to ethics committee based on their subjects.

At the capacity building area (Table 2), although most indicators have been achieved to their predictable measure as the empowerment of new groups is necessary, they should continuously be monitored parallel with the dynamic extracted needs of knowledge production indicators.

Table 2:

Under monitoring obtained achievements of capacity building axes

Capacity building indicators		Predetermined standard out put
Training workshops	The number of workshops held for academic members and other researchers	The annual trend of held workshops is ascending (from less than 300 to about 1500)and they are going to more qualified and specific subject whit more provincial or regional target groups
	The number of workshops held for students
Congresses	national congresses	The annual trend of national /International congresses is ascending and they are going to more qualified and specific subject
	International congresses
Prizes achieved in Razi and Kharazmi international Festivals	The number of awarded researchers or health research system sectors	The annual trend of awards is ascending and has almost triple in this period

Goals that must be followed more closely:

Most of the goals in knowledge production domain had a significant grow trend but they should be closely follow. The total number of published medical articles has reached from 2372 to 10,514. The growing rate of indexed articles is very high so that at the end of 2010, 95% of health related article are publishing in at least one of the indexed sites.

The indexed articles in ISI/Thomson has risen from about 533 in 2002 to 3984 in 2010 this was similarly for PUBMED indexed paper 366 up to 3638, and for Scopus 733 up to 6546.

Also indexed papers in international databases per academic member proportion rose from 0.36 in 2002 to 0.82 in 2010. Figure 1 shows the trend of the proportion of research projects ISI/Pub Med indexed articles, and in textbook cited published article to academic members during 2002–2010.

Fig. 1:

The trend of the proportion of research projects ISI/Pub Med indexed articles, and in textbook cited published article to academic members during 2002–2010

The figure shows that although during last decade the proportion of research projects and ISI/Pub Med indexed articles per academic members have a significant growth (respectively 112.5% and 433%) the trend of these indicators require to more attention. Also situations of published article that are cited in textbook are not satisfaction yet. Data also revealed the ascending growth of published articles in Iranian journals (1682 to 5835); in 2002, only one Iranian medical journal was indexed in ISI/WOS and it had no impact factor. But in 2010, 22 medical journals from Iran were indexed in ISI/Web of Science, 12 of which had impact factor. The mean impact factor of Iranian published articles in ISI/WOS was less than 0.30 rose to 2.5 in 2010.

The article presentation in national and international congresses rose from 3671 to 17325. The comparison of compilation books based on domestic researches showed a 448% growth rate (47 up to 258). The number of approved innovations and inventions increased from 10 to 2360 during the mentioned period.

Discussion

Based on Iran by 2025 defined vision, we must to receive the first grade of science position according to the ESI (Essential Science Indicator), in counterpart with other 25 south western Asian countries (14–19). As there is no any well-developed HRS based model for health knowledge promotion, strengthening and developing a national situational based model is essential for health improvement in developing countries (21–24).

In developing countries stewardship and capacity building have the essential role in health research promotion and knowledge production (20, 25, 26). As it could be seen in Table 1 and 2 nearly all of the indicators of stewardship and capacity building axes are received to their predefined levels. Moreover all of the medical sciences universities have strategic plannings which are extracted from national visions of Iran by 2025.

One of the main mission of HRS evaluation is the benefiting from stewardship for promoting the other function of HRS specially knowledge production (15, 18). Scientific output of Iran has made remarkable progress in recent years. Iran starting from a lower point with roughly 1,300 papers from 2000, with a notable surge after 2004, approached 15,000 papers by 2009 (27).

According to the website in ESI, Iran in the year 2005 and the first four months of 2009 ranked 19th in the world’s scientific output (28). Despite of such a remarkable growing, based on health research vision, we must to reach to at least 20,000 indexed articles and 280 patents registration annually. It should be noted that evaluation system could be more effective in increasing the quality and quantity health research outputs (29–31).

Our proposed model is a modified version of a more detailed WHO ranking method, Health Research System Analysis. During the last decade it has a developmental process based on our health research policy that should enable us to reach to our health research visions. Moreover for more efficacies the evaluation findings help us to provide some other supportive research resources and facilities such as:

• National and regional scientific and financial support of top level Medical universities and research centers;

• Creation of different system to encourage the researchers (Razi Festival, grant allocation, …);

• Increase in research budget allocated (from governmental to regional level);

• Capacity building for researchers and students (national and sub national workshops and training courses, research students committee foundation, .);

• Increase in indexed journals;

• Reinforcement on national journal promotion for ISI/ISI/WOS/PubMed indexing approval;

• Reinforcement of documentation;

• Revision of some related instructions and protocol (instructions of academic members scientific promotion);

• Facilitation of research policies communication;

• Facilitation of research centers approval;

• Foundation and support of research networks;

Now we are going to promote the present evaluation system to a comprehensive online evaluation system covers both our national vision as well as our regional and international research competition views.

Ethical considerations

Ethical issues (Including plagiarism, Informed Consent, misconduct, data fabrication and/or falsification, double publication and/or submission, redundancy, etc) have been completely observed by the authors.