Project Monitoring and Evaluation: An Enhancing Method for Health Research System Management

Abstract

Background:

Planning, organizing, staffing, leading and monitoring are the basic functional component of management. In present article, we aim to define the project monitoring and evaluation in health research system (HRS) considering its success and challenges based on our national experience.

Methods:

In this study based on the information of annual Medical Science Universities evaluation during the last decade the HRS indicators have been scored in three axes based on HRS functions: Stewardship, capacity building and knowledge production. In this article, we will focus on the results of HRS evaluation from 2002 to 2010, also on its success and challenges.

Results:

In an overall view, the main results are the experiences of the designing and implantation of such process after pre-project preparation, all parts followed under the whole supervision of the aims of the HRS evaluation. Project management light the way of practical application of knowledge, skills, tools and techniques for better HRS evaluation and management.

Conclusions:

We concluded that; although monitoring and evaluation as an essential part of HRS Management light the improvement ahead way but we still need to advantage of the new project management advances.

INTRODUCTION

Health research system (HRS) is defined as “The people, institutions and activities whose primary purpose is to generate the high quality knowledge that can be used to promote, restore, and/or maintain the health status of populations. It can include the mechanisms adopted to encourage the utilization of research."[1,2]

Considering the above HRS is a broad term that covers many types of researches. It is multidimensional and can be categorized in many ways. In Iran, we benefit from it as a system for planning, coordinating, monitoring and managing health research resources and activities. Another important application of HRS is in the field of promoting research to achieve the effective and equitable national health development.[1,2] We are following knowledge based development through stewardship, capacity building and knowledge production. HRS provides the reasonable bases for health needs assessment, priority setting, strategic planning and resource allocation that should be conducted constantly, through the science and art of modern management.[1,2]

Project management through enlisting the knowledge and the skills of the modern management provide the unique practical opportunity through which the processes of starting, planning, execution, controlling and closing facilitate the monitoring and the evaluation of the HRS.[3]

Indeed, such critical cases, as a national macro multidimensional health project, may only be provided through project management process, when the monitoring of the progress actions under a participatory supervision of Ministry of Health and Medical Education (MOHME) lead to corrective actions of the HRS.[4,5]

Aim to that MOHME of Islamic Republic of Iran complemented a vast multidimensional project for monitoring and evaluation of health research in Iranian Medical Science Universities, as the main elements of HRS.[6] Present paper presents the national experience of project management in the field of the monitoring and evaluation in HRS and discuss about its success and challenges.

METHODS

Aim to direction the HRS with national medium and long term developmental programs of health research; in 2001 the MOHME of Iran designed and implements the annual processes to evaluate of HRS function in medical science and their affiliated research institutions.[7,8]

In present paper, along with the technical experiences of project management in the field of HRS evaluation, we will point to some quantitative results of HRS evaluation from 2002 to 2010, also on its success and challenges.

First of all we defined the project management process in five main phases [Diagram 1]. After that each of them was followed and completed by correspond key activities. In the first stage based on the result of situation analyzes, the infrastructure such as require human source were defined. Following that the matching the objectives with the national medium and long-term developmental programs of health research lead to detailed work plan through witch our main force focused on the main tasks of data gathering, documents reviews and report writings of evaluation results. Finally, at the end, this process leads to the annual reports and comparative results.

Diagram 1.

The evaluation project management process

We retrieve the information from the national information that gathered through Medical Science Universities evaluation during the last decade. The evaluation was carried out by an expert team at undersecretary for research office. The evaluation process was reviewed and renewed annually, based on certain policies and a number of indicators were developed in line with stakeholders’ views. HRS indicators have been scored in three axes based on HRS functions: Stewardship, capacity building and knowledge production. Evaluation forms, guideline and results disseminate through MOHME website.

As one of the main input in addition to research budget; we used adjusted researcher that calculated through this formula:[6]

Adjusted Researchers = Researdh acadmic members + Educational acadmic members/3 + students of MD, DDS, PharmD/12 + Master student/3 + Residents/3 + Felowships and Ph.D students/2 + Reasearch centers researchers (non acadmic members).

In fact, adjusted researchers represent the allocated time to direct research activities by actual academic researchers of medical sciences universities and their affiliated research centers.

According to the HRS functions; specific criteria has designed in order to stewardship, capacity building and knowledge production evaluation.[9] Stewardship containing priority setting and following Iran's health innovation and science development plan by 2025; Capacity building containing holding congress and getting awards; Knowledge production containing patents and scientific publishing (book, articles and scientific presentation in congress). As a developing country, we needs organize research context to access HRS output. At this step, publication is a measurable output that discussed in present article.

RESULTS

In an overall view, the main results is the experiences of the designing and implantation of such process after pre project preparation, all parts followed under the whole supervision of the aims of the HRS evaluation.

Considering the aim and scope of the project of national evaluation of HRS, the HRS functions set as stewardship, financing, creating and sustaining resources and producing and using research.

Monitoring and evaluation of the HRS represents one of the elements of stewardship function that is tasked of monitoring and evaluation division of MOHME. In this regards, financing function and creating and sustaining resources as two functions of HRSs measured and monitored by this division. Figure 1 shows the trend of total HRS budget and health research projects budget in Medical Science Universities Furthermore the trends of Researchers and academic members.

Figure 1.

The trend of two main input of health research system in Iran during the past decade. (a) Health research system budget (b) health research system human resource

Among evaluation output, outcome and impact could be evaluated. As the main goals of health research are the improvement of scientific knowledge and utilization of knowledge to improve the health and health equity, knowledge products such as published articles have been evaluated as HRS output.

Figure 2 shows the trend of proportion of biomedical and Institute for Scientific Information (ISI) indexed articles to researchers.

Figure 2.

Trend of the proportion of the number of biomedical and ISI-indexed articles to the number of researchers

Another HRS output is presented articles in international and national congress. The result of evaluation of this output showed in Figure 3.

Figure 3.

Proportion of articles presented in national and international congresses the number of researchers

Patents and health guidelines developments are the other knowledge products that considered in HRS evaluation. As the criteria for definition and evaluation of mentioned indexes were changed during the past year, we cannot present their trends.

DISCUSSION

HRS as a multidimensional for planning, coordinating, monitoring and managing health research resources and activities provide the scientific validated research outputs.[10,11,12] It facilitates the translation and communication between the research results and health policy, health practice, and public opinion. On the other hand, it could promote the use of research to develop drugs, vaccines, devices and other health improvement applications.[10,12,13]

Based on the Health System Research in Iran by 2025 defined vision, all of the health policy makers and stakeholders especially in the field of the health research should be adopt with the systematic scientific efficient approaches. Through that they would be able to achieve to the predefined goals.[14] Project management light the way of practical application of knowledge, skills, tools and techniques for better HRS evaluation and management.[15]

One of the most important achievements of HRS evaluation is the attraction of the health policy makers’ attentions to monitoring and control of HRS budget and professional human resource.[16]

Trends of HRS input provide the evidence based information for policy makers but various trend of mentioned output required to further analyses. As shown in Figure 1, the HRS budget has followed from the ascending trend, yet the difference interval between total HRS budget and allocated budget to health project is considerable. Especially in recent year mentioned deviation is increasing and need to more political attention. Furthermore, professional human resource has increasing trend that compatible by national strategic plan 2025.[16,17]

Another success of HRS evaluation is evaluation the professional knowledge production. Figure 2 shows the difference between the trend of proportion of biomedical and ISI indexed articles to the researchers. This report during the past year, motivate the policy makers to index domestic journals in international indexing databases.

Scientific lectures and presentation in professional congress help to knowledge exchange and esteem.[18] Figure 3 demonstrates that the proportion of the presented articles in national congress to the number of researchers has been close to one but international presentation of the research results is still not satisfactory and required to more policies attention.

Result of HRS evaluation shows some validated research outputs, but impact the research outputs was not evaluated yet. It is important that the evaluation system provide the response of the main questions; what did HRS do? What is HRS impact? Where to next?[10,12,13]

In our national experience weakness of HRS outcome and impact evaluation is essential challenge that must be reviewed through national macro management of health. In this regard, professional research products citations and applied health guidelines could be considered as the operational strategies.[19]

We faced with some limitation in this study. This is undeniable truth, in addition to HRS evaluation and policy making in line it, other factors such as researchers interest, local motivators, more access to scientific resource and networks effect on HRS outputs.

In Quacquarelli Symonds World University Rankings, Research quality was assessed by Academic Peer Review and Citations per faculty criteria.[20] We can benefit from these evaluation criteria.

University ranking in China put more weight on research quantity rather than research quality. Recently they shifted from “quantity to quality."[21,22]

Now we are going to try some new techniques for monitoring and control. Earned value is a way of measuring overall performance not individual task. Earned value management is a management method for integrating scope, schedule and resources for measuring project performance and progress. The critical ratio is another method witch calculated in this way.

If critical ratio is one everything is almost certainly on target. However, the further away from one show need to investigate. HRS could be benefit from these methods in project management.[9,10,11]

CONCLUSIONS

Finally we concluded that; although monitoring and evaluation as an essential part of HRS management light the improvement ahead way but we still need to take the advantages of the new project management advances.