How to Effectively Monitor and Evaluate NCD Programmes in India

Abstract

Program monitoring and evaluation (M and E) are important components of any program and are critical to sound strategic planning. The Ministry of Health and Family Welfare, Government of India, launched the National Programme for Prevention and Control of Cancer, Diabetes, Cardio-vascular diseases and Stroke (NPCDCS) with the objectives to prevent and control common noncommunicable diseases (NCDs) through behaviour and lifestyle changes, and to provide early diagnosis and management of common NCDs. M and E of program requires identification of indicators that measure inputs, process, outputs, and outcomes. The frequency of collecting information for these indicators will vary depending on the level of use and type of indicator as well as time interval over which we expect to see a change in that parameter. A group of indicators for different domains in the three major strategies has been proposed. For effective monitoring and evaluation of NPCDCS, the way forward is to finalize the list of indicators; evolve sustainable systems for surveillance; collect baseline assessment of the indicators at district level; fix targets for each indicator for different time frames; periodic review at state and national level for monitoring progress; and establish external review mechanisms. Monitoring and evaluation require complex set of co-ordinated action, responsibility for which has to be taken up by the NCD Cell within the Ministries of Health at state and national level. However, the routine data collection and compilation could be the responsibility of Central Bureau of Health Intelligence. Integrated population-based surveys with existing disease and behaviour surveillance could be undertaken by National Centre for Disease Control. The national NCD cell should compile all these information into a meaningful policy brief so that appropriate programmatic interventions can be identified. The launch of a national program to tackle the burden of NCDs is just the beginning, and the final success of the program will depend on how effectively we monitor and evaluate it.

Introduction

Program monitoring and evaluation (M and E) are important components of a program and are critical to sound strategic planning. Monitoring refers to the simple description, counting, and tracking of processes or events. Monitoring answers the questions What? Where? When? and How much or how many? but not Is it effective? nor, Why is it effective? These questions are answered through a program evaluation. Thus, monitoring is important for assessing if something is being done and if it is being done as intended. Monitoring includes domains of program implementation and management, processes, program performance measured in terms of achievement of goals and objectives, and resource requirements (i.e., staffing and cost).

M and E of program requires identification of different indicators. These indicators would measure inputs, process, outputs, and outcomes. Input indicators measure resources devoted to a particular program or intervention. Process indicators measure ways in which program services and goods are provided. Output indicators measure the quantity of goods and services produced and the efficiency of production (i.e., number of people served). Outcome indicators measure the broader results achieved through the provision of goods and services. Impact indicators are the long-term population level changes in diseases and their outcomes. In the context of NCDs, we can define them as given in Table 1. Computation of an indicator requires information on its specific numerators and denominators which are often referred to a “data elements” in M and E terminology. The reporting formats used in various health programs are a compilation of essential data elements required for calculating specific M and E indicators.

Table 1.

Illustration of monitoring and evaluation framework for a noncommunicable disease prevention and control program

The frequency of collecting information for these indicators will vary depending on the level of use and type of indicator as well as time interval over which we expect to see a change in that indicator. Data for input and process indicators need to be collected at monthly or quarterly intervals. This will enable the program planners to assess the performance of the program in its initial phases and take corrective steps if needed. Data for outcome indictors or impact indicators (e.g. reduction of premature mortality due to NCDs, prevalence of risk factors) can be collected biennially or five yearly, as these indicators are not expected to change rapidly.

The development of an M and E framework includes an understanding of utility as per hierarchy of indicators, i.e., type of indicators required at various levels of health system for effective M and E of the program. At higher levels of health system (state, national), information on outcome and impact of program is most important, while at lower levels (block, district), the input, process, and output indicators along with the data elements are required for the program management. In order to plan for M and E of the new program, we need to understand the strategies and components of the program and review existing international experience with NCD programs and national experience with other health programs. These are discussed in the next three segments.

National Programme for Prevention and Control of Cancer, Diabetes, Cardio-vascular diseases and Stroke (NPCDCS)

The Ministry of Health and Family Welfare, Government of India, launched the National Programme for Prevention and Control of Cancer, Diabetes, Cardio-vascular diseases and Stroke (NPCDCS). Its objectives are to:

1. Prevent and control common NCDs through behavior and lifestyle changes.

2. Provide early diagnosis and management of common NCDs.

3. Build capacity at various levels of health care for prevention, diagnosis, and treatment of common NCDs

4. Train human resource within the public health setup, viz., doctors, paramedics, and nursing staff to cope with the increasing burden of NCDs.

5. Establish and develop capacity for palliative and rehabilitative care.

In the program, it is envisaged in providing preventive, promotive, curative, and supportive services (core and integrated services) for Cancer, Diabetes, Cardio-Vascular Diseases (CVD) and Stroke at various government health facilities with provisions for expanding the diseases covered under the program to chronic lung diseases, geriatric diseases, etc. The package of services would depend on the level of health facility and may vary from facility to facility. The range of services will include health promotion, psycho-social counselling, management (out-and-in-patient), day care services, home-based care, and palliative care as well as referral for specialized services as needed. Linkages of District Hospitals to private laboratories and NGOs will help to provide the additional components of continuum of care and support for outreach services. The district will be linked to tertiary cancer care health facilities for providing comprehensive care. Health education program that promotes exercise, weight reduction, screening, and early diagnosis are some of the key interventions that need to be promoted at various levels of heath facilities. These have been captured in Figure 1.

Figure 1.

Services available under NPCDCS at different levels

International Experience with M and E of NCD Programs

A number of interventions and programs in research mode or in demonstration projects from developed and developing countries have used different types of indicators to demonstrate success; however, there is paucity of this information on countrywide NCD control programs especially from developing nations. The major reason for this is that these programs are in nascent stages in most of the developing nations. However, literature search shows that the commonest indicators that have been used in setting targets and for program evaluations focus on outcomes defined in terms of favorable changes in risk factor prevalence of exercise (Japan: Target was to increase by 2010% of persons who regularly exercise to over 63%), stress, smoking [serial national representative surveys in Singapore have shown decline in smokers 17.8 (1992) to 15% (1998) to 12.5% (2004)], alcohol consumption (Mauritius: frequent alcohol consumption decreased in men from 18.2% to 14.4%), salt intake (Japan: target was to decrease salt intake in adults to below 10 g/day by 2010, midterm evaluation in 2006 showed it had come to 11.2 g from 13.5 g at baseline in 2002), hypertension, and dyslipdemias [Mauritius: reductions in total cholesterol 5.52 (1987) to 4.75 mmol/l (1992),(1–6) Japan also uses outcome indicators based on health promotion (knowledge about the risk factors, e.g., by 2010, increase to 100% people who know about the harmful effects of smoking)] as a targets for monitoring the program. Some nations have also used impact indicators such as reduction in incidence of tobacco-related cancers (Vietnam: target was to reduce by 2010, the incidence of tobacco related cancers by 30%) and adherence to diabetes treatment (Japan: target was to achieve 100% adherence to diabetes treatment by 2010) for program evaluation.(2) Input and process indicators are also vital for monitoring of any program; however, this information is more for local action and is not usually published. The draft resolution of political declaration of the High-Level Meeting of the General Assembly on the Prevention and Control of Noncommunicable Diseases, during the 66th session of United Nations general assembly has called for establishing a comprehensive global monitoring framework by end of 2012, strengthening country-level surveillance and monitoring systems, and developing global and country targets and indicators for NCD program.(7)

National Experiences with M and E of Health programs

The Indian health programs are a complex mix where independent vertical programs such as for tuberculosis and HIV/AIDS coexist with the National Rural Health Mission (NRHM). Each health program has its own system of monitoring and evaluation with its defined aims and objectives. The statistics of performance of these programs are regularly reported in reports published by the respective program management units and use used for their planning as well. In addition, the Central Bureau of Health Intelligence (CBHI) and Ministry of Health and Family Welfare also publish their annual reports which capture the essence of the achievements of these programs. At lower levels, under the NRHM, each district is expected to develop an annual District Health Plan using the information on various health indicators available at the district level. Within NRHM, in order to improve the timeliness of reporting and use of information, a strong push toward migrating the older paper-based reporting system to web-based systems has been made with the government also setting up the online HMIS portal (http://nrhm-mis.nic.in/center.aspx). Over the past two decades, information collected from independent surveys such as the NFHS and DLHS has been used to triangulate indicators and assess quality of programmatic reports. Thus, there is a strong commitment on part of the government to strengthen M and E in Indian Health Programs and the same needs to be made an important component of the NPCDCS.

The M and E for NCDCS needs to be done at all levels. However, as discussed, the focus of M and E activities at various levels will vary with disaggregate data for facility level monitoring at lower levels and indicator-based data for program evaluation at higher levels. In addition, as the outcomes and impacts of this program are expected to occur over several years, and multiple sources of information are envisaged to assess them, at the highest level, there is need to identify a health intelligence unit which can triangulate and coordinate evaluation activities. The could be similar to the Health System Evaluation Unit (HSEU) proposed by the recent High Level Expert Group on Universal Health Coverage in India for independent evaluation of the performance of both public and private health services.(8)

An M and E Framework for NPCDCS

Using the principles outlined above and the shared experience, the M and E framework for NPCDCS was developed. A group of indicators for different domains in the three major strategies have been proposed. These are listed in Box 1. For NPCDCS, as CHC or Block level is the first level that major activities are envisaged, all data elements are required at this level to compute input and process indicators. This would enable in-time assessment of program performance and corrective actions to be taken. At district level, the key indicators are related to process and output indicators which can be completely accessed through routine reports. However, as NCDs are multifactorial and NCD-related services are provided in private sector, outcome indicators would need population level data. Outcome indicators at district level will be unreliable or too costly to collect. At state and national level, major focus should be only on outcome and impact indicators. For all process and output data, at state and national level, only indicators are needed and not the data elements used for calculating the indicator.

Box 1.

List of proposed input, process and output indicators for NPCDCS

In order to monitor the progress of the program, it is important to lay down targets at the outset. This needs to be done by national and state level program managers together keeping in mind the reality of the setting in which the program will be implemented. Without these targets, it will be difficult to rate the achievement of the program. In order to guide setting of targets at national level, World Health Organization Technical Working Group on Noncommunicable Disease Targets has made certain recommendations to achieve major reductions in NCDs and their risk factors by 2025 [Table 2]. Countries can make their own targets by adapting the recommendations to their situation. It is apparent that this requires availability of baseline information.

Table 2.

Targets proposed by WHO for achievement of reduction of NCDs and their risk factors by 2025(8)

The way forward

1. For effective monitoring and evaluation of NCD Program in India, the way forward is as follows:

2. Finalize the list of indicators: A list of tentative indicators and the recording and reporting system has already been proposed. These are currently in the process of being pilot tested. It is imperative that the government moves immediately to finalize this list in consultation with all the national and state level stakeholders.

3. Evolve sustainable systems for surveillance: Nonavailability of data is one of the major reasons that programs are not effectively monitored. NCDs being a multisectoral disease will require collection of information beyond health sector. We need to establish new or utilize existing systems for data collection and analysis which are aligned to other national health programs and NRHM. There would be a need to establish quality assurance procedures for all data collected be it from health facility, laboratory, or community.

4. Baseline assessment of the indicators at district level: In order to set reasonable targets, it is important to do a situational analysis. This will also help in prioritization and appropriate allocation of resources.

5. Fix targets for each indicator for different time frames: Using the targets proposed by WHO, we can adapt these to our national context. While it might be useful to have targets set for 2025, it is important to have targets set for a shorter time frame which align with our national Five Year Plans. This will also enable us to modify the strategy based on their intermediate effectiveness. Also, it is possible that India might want to focus more on certain aspects in the immediate future and postpone certain interventions for a later date.

6. Periodic meetings at state and national level for monitoring progress: While it is relatively easy to roll out new programs, their consistent and full implementation is critical to success. This can only happen if these programs are reviewed in earnest at regular intervals, feedback provided, and strategies reassessed.

7. Establish External Review mechanisms: Just as, it has been done for NRHM, it might be useful to have an external review mechanism to monitor progress in an unbiased manner.

It is clear that monitoring and evaluation requires complex set of co-ordinated action. This responsibility has to be taken up by the NCD Cell within the Ministries of Health at state and national level. However, the routine data collection and compilation can be the responsibility of CBHI. The population-based surveys could be integrated with other disease and behaviour surveillance and could be undertaken by NCDC. NCD cell should also compile all these information into a meaningful policy brief so that appropriate interventions can be identified.

In conclusion, India has initiated a national program to tackle the burden of NCDs. But it would be good to remember that this is just the beginning and the final success of the program will depend on how effectively we monitor and evaluate it.