Evaluation criteria for the district health management information systems: Lessons from the Ministry of Health, Kenya

Abstract

Background

The District Health Management Information Systems (DHMISs) were established by the Ministry of Health (MoH) in Kenya more than two decades ago. Since then, no comprehensive evaluation has been undertaken. This can partly be attributed to lack of defined criteria for evaluating them.

Objective

To propose evaluation criteria for assessing the design, implementation and impact of DHMIS in the management of the District Health System (DHS) in Kenya.

Methods

A descriptive cross-sectional study conducted in three DHSs in Kenya: Bungoma, Murang'a and Uasin Gishu districts. Data was collected through focus group discussions, key informant interviews, and documents' review. The respondents, purposely selected from the Ministry of Health headquarters and the three DHS districts, included designers, managers and end-users of the systems.

Results

A set of evaluation criteria for DHMISs was identified for each of the three phases of implementation: pre-implementation evaluation criteria (categorised as policy and objectives, technical feasibility, financial viability, political viability and administrative operability) to be applied at the design stage; concurrent implementation evaluation criteria to be applied during implementation of the new system; and post-implementation evaluation criteria (classified as internal-quality of information; external-resources and managerial support; ultimate-systems impact) to be applied after implementation of the system for at least three years.

Conclusions

In designing a DHMIS model there is need to have built-in these three sets of evaluation criteria which should be used in a phased manner. Pre-implementation evaluation criteria should be used to evaluate the system's viability before more resources are committed to it; concurrent (operational) - implementation evaluation criteria should be used to monitor the process; and post-implementation evaluation criteria should be applied to assess the system's effectiveness.

Introduction

In the early 1970's, Kenya's Ministry of Health (MoH), recognized the need to establish health information system for collection and analysis of health service data. The information generated was expected to assist in the formulation of health policies, setting of priorities and evaluation of health care programmes. Subsequently two units of the Health Management Information System (HMIS) were created in 1975: Vital Health Statistics Unit and Evaluation and Research Unit. These units were later merged to form the present Health Information System (HIS) Department¹. The Department's stated functions were: to collect, process and analyse health and management (administrative) data; to improve the quality of health data, review and modify where necessary the reporting systems to enhance the utilization of data by end-users; to train health personnel in medical record keeping and programme evaluation techniques; and to provide other departments within the MoH with information necessary for planning purposes². It had three sections: Computing; Statistical and Medical Records, and District Health Management Information System (DHMIS)¹.

The main function of the DHMIS Section was to initiate and coordinate the development of District Health System (DHS). Prior to the introduction of this system, health facilities collected information haphazardly and irregularly, the data collected was incomplete and unreliable for use at the point of collection, and the volume of data collected was too large rendering analysis impossible². Although much administrative- and health programs-related data was collected at the district level, little use was made of it at that level. The data was merely collected and compiled by the health information personnel, then submitted to the MoH headquarters with little or no feedback provided back to the districts. It soon became clear that data collected on routine basis was not useful for planning and evaluation of district level health services. The challenge then was to develop an information system for the District Health Management Teams (DHMTs) that would provide health information for local use and facilitate achievement of the expected management and planning functions².

The first DHMIS in Kenya, funded by the United Nations Children's Fund (UNICEF), was piloted in Murang'a DHS in 1988.³ At the same time, a number of agencies initiated DHMISs in various districts. UNICEF, for instance, funded district health systems in four districts (Kwale, Mombasa, Baringo, Kitui) and in Mombasa Municipality⁴; the African Medical Research Foundation (AMREF) funded DHMISs in Nyamira and Nyandarua districts²; the Swedish International Development Agency (SIDA) supported the establishment of DHMISs in Kisumu and Uasin Gishu districts as well as in Kisumu Municipality; and the Finnish Government through the Kenya-Finland Primary Health Care Programme (KFPHCP) funded DHSs in Western Province⁵ . These systems later became an integral part of the health care system. A major feature of these systems was that they were all manually operated and fully donor funded.⁶.

Since the establishment of these DHMISs in the various districts and Municipalities in Kenya, there has been no formal comprehensive evaluation to assess their strengths or weaknesses. The objective of this study was therefore to propose appropriate evaluation criteria for the DHMIS that can be used to provide valid and reliable assessment of the system. It thus attempts to develop three sets of evaluation criteria applicable at all phases of the DHMS: in the design, implementation and post-implementation phases of the systems.

Methods

This was a cross-sectional study undertaken in three districts (Murang'a, Uasin Gishu and Bungoma) which had implemented DHSs in Kenya. The study employed a range of qualitative approaches, including documents review, Focus Group Discussions (FGDs) and Key Informant (KI) interviews to collect information on the development and implementation of the various District Health Management Information Systems (DHMISs). A comprehensive review of relevant literature on the subject was undertaken. The review covered records, reports and operational manuals of the implemented systems both at the MoH headquarters and at the district level of the respective DHSs. Additional literature was identified by snowballing from the references cited in the available reports and documents. A total of 9 FGDs were conducted with systems users to identify their perspectives and expectations on the aspects of the designed and implemented system that needed to be addressed in the evaluation. After obtaining approval of the Ministry of Health for the study, the Medical Officer of Health in each study district was approached and requested to provide a list of staff who had been involved in the design and implementation of the DHS. These individuals included medical records personnel, nurses, clinical officers, hospital administrators and medical officers. From each category, 2–3 purposely selected staff were invited to participate in the FGDs. Key informant interviews were also held with 3 DHMIS designers at the MoH headquarters, 12 DHMIS staff and 15 end-users in three of the DHSs that implemented these systems. An interview guide was developed and used to generate information on the following key issues: overall systems design processes, inputs and outputs (information characteristics - relevance, completeness, accuracy, timeliness, format); quality and quantity of system personnel, status of processing equipment, availability of working and storage space; systems cost-effectiveness, and system impact on health delivery. Respondents were interviewed by the PI (GWO) at their respective places of work. Each interview lasted approximately 45 minutes. All interviews were tape-recorded and later transcribed. Content analysis of the printed texts was done manually to group the emerging themes and issues mentioned by the respondents with respect to evaluation of their system. Summaries of narrative descriptions of their perceptions on the desired evaluation criteria were recorded under each phase of implementation of the DHMISs.

Results

From the documents reviewed, it was established that the first DHMIS was implemented in Murang'a in Central Province in 1988, funded by UNICEF. It had a total of 26 data collection forms, 15 of which were used to collect administrative information and the remaining 11 were used for collecting health information.³ The Bungoma DHMIS in Western Province was started in 1989 with funds from the Finnish Government⁵. Three years later, in 1992, the Swedish Government funded the Uasin Gishu DHMIS in Rift Valley Province; it had a total of 49 data collection tools.⁷

Information gathered from the interviews indicated that the DMHISs implemented did not incorporate evaluation aspects at their design stage. The respondents felt that there was need to come up with evaluation criteria for DHMISs. The respondents' justification for establishing evaluation criteria at the system design stage was that by stating the criteria in advance, the designers of the system would be able to set guidelines for monitoring performance of the DHMIS. They also pointed out that by stating evaluation criteria early, designers will avoid the temptation to rationalise the introduced DHMIS. Based on the information from the interviews and discussions, three sets of DHMIS evaluation criteria were identified: pre-implementation, concurrent-implementation (operational) and post-implementation evaluation criteria (Table 1):. A detailed description of these criteria is presented below.

Table 1.

Evaluation criteria

Pre-implementation		Concurrent (operational)	Post-implementation
•	Policy and Objectives	Achievement of Objectives	Internal (quality of information produced)
•	Technical Feasibility	Technical Capacity (Trained Personnel, adequacy and availability at right places)	External (resources and managerial support)
•	Financial Viability	Financial status - flow of funds for operations and sustainability
•	Political Viability	Sustained political support
•	Administrative Operatibility	Compatibility with DHS's activities	Ultimate (impact)

Pre-implementation evaluation criteria

Due to various problems faced by operating the implemented DHMISs in the DHSs, the respondents felt that for every new DHMIS, there was need to assess its viability before it was implemented. This necessitated the formulation of pre-implementation evaluation criteria. This set of evaluation criteria, to be used in the first phase of the implementation, looks at the issues that need to be addressed before committing the scarce resources to the establishment of a DHMIS. In the first phase (planning and design stages) of a DHMIS project, this evaluation focuses on the scope and detail of data sets to be collected. In order to ascertain the viability of such a potentially costly endeavour, the respondents emphasised the need to incorporate evaluation criteria at the design stage of every DHMIS. Their views can be categorised under the following themes: policy and objectives, technical feasibility, financial viability, political viability and administrative operability.

▪ Policy and objectives

The policy and objectives criterion seeks to find out whether the organisation designing and implementing the DHMIS has developed and documented relevant policy and objectives for the proposed information system. This criterion addresses policy issues concerning the system's activities such as information systems development (compatibility issues), information system standards, data collection strategy (public and private health sector partnership), information processing (manual, computerised or a mix of both), dissemination (confidentiality) and storage (security). This criterion also addresses DHMIS objectives such as its ability to: determine end-user information needs; facilitate data gathering and information processing; and ability to improve quality of DHS management through provision of the right information to the right user at the right time.

▪ Technical feasibility

Technical feasibility criterion requires an assessment of information technology (IT) infrastructure. IT infrastructure can be divided into two related but distinct components - a technical and a human infrastructure. Technical infrastructure is a set of shared, tangible IT resources forming a foundation for business applications (hardware, software and data). Human infrastructure includes human and organizational skills, expertise, knowledge, commitments, values, and norms. Availability and adequacy of both technical and human infrastructure must be ascertained up-front.

▪ Financial viability

The key question that needs to be answered is: Are the necessary resources available? If not, what is the likelihood that they will be available when the DHMIS is ultimately implemented? If financial implications of a DHMIS are clearly stated, it makes it possible for decision-makers to see how much of the available scarce resources can be committed to it. This criterion requires that financial viability be ascertained for purposes of sustainability of the DHMIS.

▪ Political viability

Political viability criterion addresses the likely outcomes of the DHMIS in terms of support from relevant power groups such as legislators, decision-makers, administrators, and system users. Measurements in this category are often subjective and less quantifiable. Policy is normally developed in the political arena and must survive the political test. Consequently, introduction of a DHMIS in the DHS must be subjected to political assessment. Political criterion calls for the consideration of issues such as acceptability, appropriateness, responsiveness, legality and equity. Acceptability refers both to the determination of whether the DHMIS will be acceptable to actors in the political process, and to the determination of whether clients and other actors will be receptive to the DHMIS. Appropriateness is related to acceptability in that it addresses the issue of whether the DHMIS's objectives will mesh with the values of the user-community or the DHS managers. Responsiveness is related to acceptability and appropriateness and involves the target group's (DHS managers') perception of whether the DHMIS will meet its needs. A DHMIS might be efficient and effective but not necessarily match with what the DHS managers need or want.

▪ Administrative operability

This criterion focuses on how possible it will be to actually implement a DHMIS within the political, social and most important administrative context. If the administrative talent and delivery system are not available, then the operability of the DHMIS must be called into question. Specific issues to consider in evaluating administrative operability include authority, institutional commitment, capability and organisational support. Authority to implement a DHMIS to turn it into a working system requires institutional commitment from both above and below. Capability, both staff and financial, is essential to DHMIS implementation. Answers must be provided to the following questions: Do administrators have the necessary knowledge and skills to put the DHMIS into effect? Does the implementing DHS have the financial capacity to do what will be demanded of it? Organisational support is also an important criterion, because it is not just sufficient to have only the authority to implement a DHMIS and the commitment of key personnel. It is also necessary to have appropriate and adequate equipment, physical facilities and other support services.

The concurrent (operational) - implementation evaluation criterion

This criterion basically monitors the actual status of each of the specific criteria identified at the pre-implementation stage, as envisioned by the designers and sponsors of the system. At the implementation stage, evaluating the operations of the DHMIS can help identify faults and deficiencies in both the design and implementation strategy. For successful implementation of a DHMIS, it is necessary to carry out evaluation during implementation as this offers the possibility to adapt to changing circumstances. This criterion also aims at determining what aspects of the implemented DHMIS need reviewing. It is, thus, essential for identifying both what is possible under existing conditions and what changes are needed to facilitate successful implementation of the DHMIS.

Post-implementation evaluation criteria

The third and final phase, the post-implementation evaluation criteria focus on ascertaining the extent to which the hypothesized benefits of the introduced system have been achieved. Issues such as impact of the DHMIS on DHS management are addressed. Based on the interviews and discussion, respondents classified post-implementation evaluation criteria for the DHMIS into three main categories: internal, external, and ultimate criteria.

Internal criterion

Internal criterion relates primarily to the quality and utility of the information itself. The two requirements are closely related. Internal criterion for evaluating a DHMIS needs to be related to overall system design - system inputs, processing and outputs. The internal criterion addresses specific information characteristics such as: completeness (comprehensiveness), timeliness, reliability, and operability. Improving a DHMIS in these respects is a task, which calls for a variety of technical skills and knowledge from many fields. Information relevance, accuracy, completeness and timeliness determine its quality.

▪ External criterion

External criterion relates to those issues, mostly non-system properties, which affect proper operations, utilization and acceptance of the DHMIS and its products. This criterion addresses issues related to continued existence of adequate resources, managerial support, skilled DHMIS personnel and DHMIS-oriented medical personnel. Without resources and managerial support the sustainability and ultimate survival of DHMIS's can not be guaranteed.

▪ Ultimate criterion

Respondents stressed that at the end of the implementation, there is need to assess the overall impact of the DHMIS on the DHS and on service recipients and the protection of patients' right to privacy of personal information obtained in the process of care. There is need to have a criterion for assessing such impact. The ultimate criterion is thus a special type of external criterion, which is applied in making judgements as to the worth or value of the entire DHMIS enterprise.

Discussion

From the results, it is clear that there was no documentation of the policy and objectives of each of the implemented DHMISs in Kenya. In the absence of clear policy and objectives to guide the design and implementation, the various DHSs have developed different models (versions) rendering compatibility and comparison impossible.^6,7 Furthermore, the systems were characterized by lack of integration and effective central coordination to ensure that the information gathered was shared by other health care providers in the same health care system.⁸ Most of the health personnel interviewed had no idea on the policy and objectives of the DHMIS. This was further complicated by the fact that these systems were conceptualised and designed at the MoH headquarters, and merely implemented in the DHSs without the involvement of the DHS staff in their design. Health policy makers often assume that they can design a DHMIS, someone can simply implement and manage it. Implementation of DHMIS requires change in staff behaviour, and without it, the many seemingly small actions necessary to implement it may not be carried out. Our findings are consistent with a previous report showing that the DHS managers were not fully involved at the design stage, and were not supportive of the DHMIS's activities⁶.

Political criterion is closely related to policy criterion. It requires that key stakeholders, such as the Minister for Health and the Director of Medical Services, be sensitized about the DHMIS and provide the required support. In addition, health care management is changing continuously and senior administrators in the Ministry often change responsibilities. This means that the teams overseeing the design, implementation and operation of such projects are rarely in position for the entire period of implementation⁹. This situation calls for proper documentation of every decision and activity so that incoming policy makers can follow the developments. Political criterion also includes the acceptability of the information system to policy makers. Without the full support of these key personalities, sustainability of such initiatives becomes a major challenge. Equally important is the issue of end-user involvement at both the design and implementation stages¹⁰. The MoH does not appear to have given this criterion serious thought it deserved.⁶ Any proposed DHMIS needs to be subjected to a rigorous pre-implementation evaluation before the available scarce resources are committed. This requires that pre-implementation evaluation criteria be developed as part and parcel of the design of a DHMIS.

Similarly, financial viability criterion was not adequately addressed in all the systems studied. Given the fact that all implemented DHMISs were donor funded, and since this source of funding was bound to come to an end, there should have been serious consideration about sustainability at the expiry of donor support. None of the DHMISs had a recurrent budget from the exchequer for its operations. There was no evidence of any budgetary estimates by the MoH, on the costs of design, implementation and sustainability of the DHMISs. In addition, failure to assess availability and adequacy of both technical and human infrastructure up-front resulted in several operational problems, such as untrained personnel being involved in the DHMIS activities hence compromising the quality of the information generated. At the same time, the introduction of the systems' numerous data collection forms without involving the DHS personnel, created an additional workload for staff. As reported elsewhere⁶, this new requirement was resisted by staff resulting in improper or non-use of the introduced forms, and eventual collapse of the systems.

Quality improvement criterion revolves around the implementation of quality assurance programme for the information system and needs to be incorporated in the implementation process¹¹. Since one of the main objectives of a DHMIS is to provide accurate information to enhance quality of health service delivery, efforts must be made to ensure that the data collected meets the DHS requirements. For this to be achieved, certain level of training, resources and managerial capacity are required. However, this study revealed that most of the DHMIS personnel have not been adequately trained on data collection and processing. In addition, the lack of adequate resources and managerial support further posed obstacles to the systems' operations.

For effective implementation of any DHMIS, it is prudent to apply the external evaluation criterion in order to ascertain the degree to which the DHS management is supportive of the DHMIS's operations. This factor is based on the premise that any information system cannot function effectively unless the people who develop, manage and use it are adequately trained, and users have developed an ‘information culture’ which enables them to appreciate the value of information as a resource for effective health service delivery.^10,12 Achievement of a DHMIS objectives depends on availability of resources for its development and operation, functioning of equipment, security and confidentiality of data, availability of sufficient ability and motivation of providers, planners, politicians, and the public to make proper and effective use of the DHMIS. The successful operation of any DHMIS requires an accurate evaluation and adequate provision of these resources. A report by Schmitz clearly supports the need to evaluate the impact of information systems on health care delivery¹³. There is no evidence from this study to show that such an evaluation has been undertaken. Establishing the various evaluation criteria would therefore provide the tools for assessing whether the DHMIS is producing the expected results or not.

Conclusion

The lack of evaluation criteria presents a major shortfall in the design, implementation and operation of DHMIS. This study identifies evaluation criteria which can be applied to assess the development, implementation and outcomes of district a DHMIS. There is need to incorporate basic sets of evaluation criteria at the design stage of HMIS. Pre-implementation evaluation criteria should be used to evaluate the system's viability prior to committing resources to its implementation; concurrent evaluation criteria should be used to monitor the process of implementation; and finally, post-implementation evaluation criteria need to be applied to assess the system's effectiveness in the management of district health services.