Monitoring iCCM: a feasibility study of the indicator guide for monitoring and evaluating integrated community case management

Abstract

Most countries in sub-Saharan Africa have now adopted integrated community case management (iCCM) of common childhood illnesses as a strategy to improve child health. In March 2014, the iCCM Task Force published an Indicator Guide for Monitoring and Evaluating iCCM: a ‘menu’ of recommended indicators with globally agreed definitions and methodology, to guide countries in developing robust iCCM monitoring systems. The Indicator Guide was conceived as an evolving document that would incorporate collective experience and learning as iCCM programmes themselves evolve. This article presents findings from two studies that examined the feasibility of collecting the Indicator Guide’s 18 routine monitoring indicators with the iCCM monitoring systems that countries currently have in place. We reviewed iCCM monitoring tools, protocols and reports from a purposive sample of 10 countries in sub-Saharan Africa. We developed a scorecard system to assess which of the Indicator Guide’s 18 routine monitoring indicators could be calculated with the given monitoring tools, and at which level of the health system the relevant information would be available. We found that the data needed to calculate many of the Indicator Guide’s routine monitoring indicators are already being collected through existing monitoring systems, although much of these data are only available at health facility level and not aggregated to district or national levels. Our results highlight challenge of using supervision checklists as a data source, and the need for countries to maintain accurate deployment data for CHWs and CHW supervisors. We suggest that some of the recommended indicators need revising. Routine monitoring will be more feasible, effective and efficient if iCCM programmes focus on a smaller set of high-value indicators that are easy to measure, reliably interpreted and useful both for global and national stakeholders and for frontline health workers themselves.

Key Messages

• The iCCM Task Force has developed a list of recommended indicators for monitoring iCCM, with globally agreed definitions and methodology.

• The data needed to calculate many of these routine monitoring indicators are already being collected through existing iCCM monitoring systems, though some of the indicators, in their current form, may be overly difficult to measure and need revising.

• Routine monitoring of iCCM will be more feasible, effective and efficient if countries are encouraged to measure a smaller set of high-value indicators that are easy to implement, reliably interpreted and useful for both global stakeholders and frontline health workers themselves.

Introduction

Most countries in sub-Saharan Africa have now adopted integrated community case management (iCCM) of common childhood illnesses as a strategy to improve child health (Rasanathan et al. 2014). iCCM programmes involve training and equipping CHWs to classify and treat sick children at community level, typically for malaria, pneumonia, diarrhoea, malnutrition and/or newborn care (CORE Group 2010; UNICEF 2012a). WHO and UNICEF advocate iCCM of common childhood illnesses as ‘an essential strategy that can both foster equity and contribute to sustained reduction in child mortality’ (UNICEF 2012a). To ensure that iCCM achieves its potential, the implementation of iCCM programmes should be guided by robust monitoring and evaluation (M&E). However, most countries have not yet integrated iCCM M&E as part of their national Health Management Information Systems (HMIS). At the 2014, iCCM Evidence Review Symposium, ministries of health, donors and partners reaffirmed the importance of monitoring and evaluating iCCM programmes to improve implementation strength and programme impact (UNICEF 2014). Valid and timely measures of iCCM implementation enable programme managers to identify problems, improve implementation and report progress to national and international stakeholders (UNICEF 2012b; Guenther et al. 2014).

In March 2014, the iCCM Task Force, an association of multilateral and bilateral agencies and non-governmental organizations (NGOs) working to promote iCCM, published an Indicator Guide for Monitoring and Evaluating iCCM (McGorman et al. 2012; MCHIP 2013b). This Indicator Guide lists recommended iCCM indicators useful across programme components and phases to ‘encourage the consistent use of standardized definitions and metrics for iCCM indicators, serve as a resource for iCCM programmes to improve M&E systems and promote improved M&E of iCCM programmes’ (MCHIP 2013b). The 48 indicators in the Indicator Guide span the eight components of the iCCM Task Force’s Benchmark Framework: coordination and policy setting, costing and financing, human resources, supply chain management, service delivery and referral, communication and social mobilization, supervision and performance quality assurance, and M&E and health management information systems (McGorman et al. 2012; MCHIP 2013a).

The Indicator Guide is not intended as a prescriptive set of indicators for all iCCM programmes, but rather as a ‘menu’ of indicators with globally agreed definitions and methodology. Ministries of health and implementing partners can use the Indicator Guide to identify the most appropriate indicators for their iCCM programme and context. The iCCM Task Force conceived the Indicator Guide as an evolving document that, in time, would incorporate collective experience and learning as national iCCM programmes themselves evolve. When the development of the Indicator Guide was initiated in 2010, few indicators had been adopted and used by iCCM programmes at national scale. Many of the Indicator Guide’s indicators were initially adapted from sub-national iCCM programmes with well-resourced M&E systems run by NGOs. Developing a set of indicators that is suitable and practical for monitoring national iCCM programmes will take time, experience and further research.

As a first step towards improving the Indicator Guide, the iCCM Task Force sought to assess the feasibility of collecting its indicators through existing M&E systems. One study, commissioned as part of the USAID-funded Translating Research into Action (TRAction) Project, was conducted in Ethiopia, Malawi, Mali and Mozambique (Hazel et al. 2014; USAID 2014). A second study was commissioned by USAID’s Maternal and Child Health Integrated Program (MCHIP) and examined the Democratic Republic of the Congo (DRC), Madagascar, Niger, Senegal, South Sudan and Zambia (MCHIP 2014). The goal in each study was to understand the opportunities and challenges for measuring the Indicator Guide’s indicators given the existing tools and systems used to monitor and evaluate iCCM. As part of this research, the studies’ authors determined which indicators, including component numerators and denominators, were already being reported in each country. In this article, we present the findings from these two studies and discuss the implications for the roll-out and continued development of the Indicator Guide.

The indicators in the Indicator Guide are grouped into three categories: routine monitoring indicators, special studies indicators and national-level milestone indicators (MCHIP 2013b). In this article, we focus on the 18 routine monitoring indicators: the indicators that are expected to be available over time at health facility, district and regional levels, and collected through an iCCM programme’s M&E system, health management information system (HMIS) or other routine sources (the definitions of these 18 indicators are provided in Table 3, Supplementary Annex S1). Routine monitoring indicators involve the development of a fixed set of tools and protocols to be adopted by hundreds or thousands of community health workers (CHWs) and other health system staff. The collection of these indicators entails a specific set of issues and challenges, and we therefore decided to analyse them as a group.

Table 3.

Characteristics of 18 routine monitoring indicators

	A	B	C	D
	Information available at health facility or below (number of countries)	Indicator requires numerator and denominator from different sources	Data collected through supervision checklists	Indicator requires CHW or CHW supervisor deployment data
3. Human resources
3.2: iCCM CHW density	3	Yes		Yes
3.3: Targeted CHWs providing iCCM	5	Yes		Yes
3.4: Annual iCCM CHW retention	6	Yes		Yes
4. Supply chain management
4.2: Medicine and diagnostic availability	10		Yes
4.3: Medicine and diagnostic continuous stock	9		Yes
4.4: Medicine and diagnostic storage	9		Yes
4.5: Medicine and diagnostic validity	10		Yes
5. Service delivery and referral
5.1: iCCM treatment rate	7	Yes
5.2: Caseload by CHW	8	Yes		Yes
5.3: Referral rate	7
7. Supervision and performance quality assurance
7.2: iCCM supervisor training	0			Yes
7.3: CHW-to-supervisor ratio	1			Yes
7.4: Routine supervision coverage	5	Yes	Yes	Yes
7.5: Clinical supervision coverage	9	Yes	Yes	Yes
7.6: Correct case management (knowledge)	9		Yes
7.7: Correct count of respiratory rate	10		Yes
7.8: Complete and consistent registration	8		Yes
8. M&E and health information systems
8.3: District reporting	2

Methods

Data collection

The two studies contributing to this article each involved a document review of iCCM tools and protocols from multiple countries. The resulting sample of 10 countries was not intended to be a representative sample of all countries implementing iCCM, but rather a purposive sample to generate initial lessons for improving the Indicator Guide. These 10 countries reflect a range of iCCM programmes at different stages of development, all with at least 5 years of national or regional implementation.

Copies of the monitoring tools used in each of the 10 countries, along with national iCCM M&E protocols, reports and other related documents, were requested from ministries of health or implementing partners. We did not have prior expectations about the tools that each country would or should be using. The Indicator Guide does not suggest an ‘ideal’ M&E system for iCCM; rather, it focuses on the data to be collected as part of the national HMIS. That said, we know from implementation experience that four types of tools are typically used to monitor iCCM: (1) tools used by CHWs to report information related to individual consultations, including sick child forms, referral forms and/or patient registers (kept at CHW level); (2) tools used to aggregate and communicate data collected by CHWs, often in the form of a monthly report for CHW supervisors (kept at health facility level); (3) tools used by CHW supervisors to record information during supervision visits; for example, data on a CHW’s drug kit, equipment, patient register and performance (kept at health facility level); and (4) tools used to aggregate and send information from health facility level to higher levels of the health system (kept at district, regional and national levels). In requesting M&E tools from ministries of health and implementing partners, we asked for the above tools (if they existed), as well as any other tools that key informants thought relevant to our analysis. All monitoring tools that we collected and reviewed had been endorsed by the ministry of health for use in national or regional iCCM programmes.

For the TRAction study, researchers travelled to Ethiopia, Malawi and Mozambique to collect documents in person and conduct follow-up interviews with 9–14 key informants in each country. Key informants for both the TRAction and MCHIP studies were staff from the ministry of health or implementing partners with responsibilities for monitoring and evaluating iCCM in their country. The information gathered through key informant interviews was used to verify our understanding of the M&E tools; for example, the intended user of the tool, the frequency with which the tool was to be completed and submitted and how data from the tool was to be aggregated. For the TRAction study in Mali, and for the MCHIP study, documents were collected remotely. Researchers wrote to two to four key informants per country and requested documents by email. Follow-up interviews were conducted by phone to ensure that we had collected all the monitoring tools being used in the country, that the tools had been endorsed by the ministry of health, and to verify our understanding of the tools and their intended use.

Analysis

Once the monitoring tools were collected and reviewed, we developed a scorecard system to assess which of the Indicator Guide’s 18 routine monitoring indicators could be calculated with the given monitoring tools, and at which level of the health system the relevant data would be available. For each of the indicators, the availability of the data required to calculate the indicator was classified on a colour scale: green, for data available at district level; yellow, for data available at health facility level; orange, for data only available in forms or records kept by CHWs; and red, for data that are not available at all. Our analysis did not address issues of data quality, data completeness or data use. Rather, we examined the feasibility of collecting the indicators with the current iCCM M&E systems in the 10 countries, assuming that the M&E systems are working as expected—in other words, assuming that CHWs and health workers are using the monitoring tools appropriately, completing the tools accurately and submitting the tools on time.

Results

We organized our findings into three topic areas: the monitoring tools used to collect routine data for the ten iCCM programmes, the availability of the information needed to calculate the indicators and the challenges associated with calculating the indicators.

Tools used to collect routine data

The monitoring tools used in the 10 countries are listed in Table 1. As this table illustrates, there are noticeable differences in the nature of the tools used in each country and in the way data are aggregated. In six countries, CHWs record details of consultations using individual sick child forms; one form per child. In the other four countries, CHWs record details of consultations directly into a patient register, with each row in the register representing one child. Half of these sick child forms and patient registers are relatively simple, with pictures indicating the information to be recorded, while the others are word-based and more complex. In six countries, CHWs compile their own monthly reports to submit to the nearest health facility; in the other countries, supervisors aggregate consultation numbers during supervision. In five of the countries, information collected during supervision visits is recorded on one checklist per CHW; in the others, it is recorded on several different checklists per CHW; for example, different checklists for drug availability, register completion, CHW performance. In DRC, supervisors use an additional checklist to collect information from child caregivers. In Ethiopia, health workers compile a monthly report of supportive supervision activities.

Table 1.

Monitoring tools collected for review, by country

	DRC	Ethiopia	Madagascar	Malawi	Mali	Mozambique	Niger	Senegal	South Sudan	Zambia	All countries
Total number of monitoring tools reviewed	8	6	5	7	4	4	3	5	4	5	51
Consultation records
Sick child reporting form (individual case form)	1		1	1	1		1	1			6
Referral form (and cross-referral)	1		1	1			1	1			5
Register of consultations	1	1		1	1	1			1	1	7
Register of medicines								1			1
Monthly reporting forms
Monthly report aggregating multiple consultations by one CHW [submitted to health facility by CHW]	1	1		1	1	1				1	6
Monthly report aggregating monthly totals for multiple CHWs [submitted to district office by health facility]	1	1	1	1		1		1	1		7
Supervision checklists
Checklist for supervision of one CHW	1	1		1	1	1	1	1	1	1	9
Checklist for register review and/or medicines review for one CHW	1		1								2
Checklist for clinical supervision (observation of consultation)		1	1	1						1	4
Interview guide for caregiver exit-interview or follow-up interview	1										1
Supportive supervision report		1									1
Checklist for supervision of CHW supervisors by district office									1	1	2

It is not the place here to talk about the merit of different tools; such issues are context-driven and require deeper analysis. But it is worth noting the variability between countries and the potential link between the type of tools used in a country and the availability and quality of information for calculating indicators. Multiple, complex tools could unnecessarily burden CHWs, CHW supervisors and the health system as a whole, requiring time and skills to aggregate, and potentially compromising data quality.

Availability of information needed to calculate indicators

The 18 routine monitoring indicators listed in the Indicator Guide were cross-checked against the tools from the 10 countries to determine the availability of the data needed to calculate each indicator. Table 2 summarizes the results of this analysis. As this table shows, countries are already collecting the data needed to calculate many of the routine monitoring indicators. The availability of data differs from country to country, but in general the indicators for which data are most available concern human resources (iCCM Benchmark Framework component 3), service delivery and referral (component 5) and M&E and health information systems (component 8). Data are less available for indicators concerning supply chain management (component 4) and supervision and performance quality assurance (component 7).

Table 2.

List of 18 routine monitoring indicators with summary colour classifications indicating the highest level of the health system at which data is available. A colour version is available online.

In most cases where data are unavailable, it is because the monitoring tools do not collect the data required to calculate the indicator. For example, supervision checklists in three countries do not capture whether the supervisor observed the CHW in consultation with a child, meaning that indicator 7.5 ‘clinical supervision coverage’ cannot be calculated. Similarly, supervision checklists in seven countries do not record whether the CHW has expired medicines, meaning that indicator 4.5 ‘medicine and diagnostic validity’ cannot be calculated. In other cases, the monitoring tool captures the type of data needed for the indicator, but not in the exact format required by the Indicator Guide; for example, indicator 4.3 ‘medicine and diagnostic continuous stock’ asks for stockouts in the past month, but in Niger the supervision checklist asks for stockouts in the past 3 months.

Challenges associated with calculating indicators

Our analysis highlighted three common challenges associated with calculating the 18 routine monitoring indicators. Table 3 shows the indicators affected by each challenge.

Extracting data from available sources

Although countries are already collecting most of the data needed for the 18 routine monitoring indicators, much of these data are only available at health facility level, not district and national levels (as shown in Table 2). The necessary data collected by CHWs are only submitted to health facilities and not sent up the chain to district offices, either because the data are not included in a health facility report, or because the data are aggregated in such a way that the relevant indicator cannot be calculated. Thus, if one wanted to calculate the indicators at the national level, a data collector would need to undertake additional work to examine, count and aggregate records kept at health facilities. (See Table 3, column A, for the number of countries per indicator for which data are only available at health facility level.) For some indicators and countries, this may not be a problem. The Indicator Guide does not specify at which level of the health system an indicator is to be reported and countries are encouraged to make those decisions for themselves (MCHIP 2013b). Data are useful for many stakeholders—service providers, policy makers, global actors—and countries may rightfully decide that certain data only need to be available at the health facility or district level; for example, to assist CHW supervisors in managing CHWs or to assist district health staff with programme management and resource allocation.

A related issue concerns the ‘usability’ of data; the work required to calculate indicators following data extraction. Some indicators require a numerator from one source and a denominator from another source; or alternatively, require computing information from multiple data points in a single tool. In some cases, a data collector would not only need to examine health facility records, but would also need to examine each record in detail to determine how to interpret the record. For example, indicator 7.8 asks for the ‘number of CHWs whose registers show completeness and consistency between classification and treatment for at least four out of five cases reviewed’. To calculate this indicator using the existing tools in eight countries, a data collector would need to analyse the classifications and treatments listed in individual supervision records kept at health facilities, determine which records are consistent in at least four of five cases, and count the total number of consistent records, before reporting the results to district level and above. As an indication of the effort required to calculate the indicators, Table 3, column B, shows the indicators that require data from two different sources or monitoring tools.

Relying on supervision checklists as a data source

Nine of the routine monitoring indicators require data that in most countries are being collected via supervision checklists (see Table 3, column C). This is problematic for several reasons. Unless a country has a robust supervision schedule, supervision checklists will likely only be completed on an ad hoc basis, so it may be the case that these indicators cannot be calculated as frequently as other indicators. Some CHWs may be supervised more frequently than other CHWs, and this may distort indicator measurements—particularly if those CHWs that are being supervised regularly perform either better or worse than other CHWs. Seven of the indicators that take data from supervision checklists use ‘CHWs assessed’ as a denominator. Unless one can be certain that all CHWs are supervised using standard methods, the results are likely to be non-representative and associated with measurement errors. Furthermore, of the 10 countries studied for this report, only two have tools for aggregating and communicating information from supervision checklists to district level, so data that are collected via supervision checklists will not be communicated beyond the health facility level.

Maintaining accurate CHW deployment data

Eight of the 18 routine monitoring indicators require the use of CHW deployment data that, in most countries, is not routinely reported through the iCCM monitoring system (see Table 3, column D). Seven indicators require information on CHW deployment or training, and two indicators require information on CHW supervisor deployment or training (one indicator requires both CHW and CHW supervisor deployment data). In order for these indicators to be calculated, a country must maintain records on CHW and CHW supervisor deployment. None of the monitoring tools that we analysed for this article had fields to report CHW deployment data. The key informants that we spoke with in all countries said that they believed these CHW deployment data were available at district level (hence the positive results for these indicators in Table 2), although the reliability and routine nature of these data sources are unclear. The number of CHWs submitting monthly reports could conceivably act as a proxy for CHW deployment, but counting submitted reports only gives information on active CHWs, not the total number of CHWs deployed. For indicators such as 3.3 ‘targeted CHWs providing iCCM’, using the number of CHWs submitting monthly reports as the denominator for the proportion of CHWs providing care would not capture those CHWs who have been deployed but are not carrying out their role—which is the purpose of the indicator.

Discussion

The results of our analysis show that, in the 10 countries studied, the data needed to calculate many of the Indicator Guide’s recommended routine monitoring indicators are already being collected through existing monitoring systems. Appropriate fields are included in consultation records, monthly reporting forms and supervision checklists, and processes are in place to aggregate and report the necessary information at health facility level and above. This is a promising finding. It affirms that countries and partners have established monitoring systems that could, in principle, measure a set of standard indicators such as those recommended by global iCCM experts in the Indicator Guide.

However, data for some of the 18 routine monitoring indicators are not being collected, and measuring these indicators would require either changes to tools and protocols or additional work on the part of health facility and/or district staff to compute indicators from data in one or more tools. In some cases, countries would need to revise CHW reporting forms or supervisor checklists; new fields would need to be added or rewritten to reflect the indicator definitions listed in the Indicator Guide. In other cases, countries would need to revise reporting templates at health facility level to ensure information is appropriately aggregated and communicated to the relevant level for action. Some indicators, such as those concerning supply chain management and performance quality assurance, may, on reflection, be too demanding for current monitoring systems, and might be better collected through special studies. Deployment data are needed as a denominator for eight indicators, including indicators on the proportion of active CHWs and the ratio of CHWs to population. Because deployment data are not typically reported in monitoring tools, calculating these indicators would require ensuring that accurate deployment data are available. Similarly, nine of the 18 routine monitoring indicators require data from supervision checklists, which, given the challenges of individual supervision of CHWs, may result in untimely or unrepresentative data (Hill et al. 2014; MRC 2014). In many countries, data from supervision checklists are not being aggregated routinely in the way that monthly reports from CHWs are aggregated and collected, and the quality of these data is subject to the capacity and work practices of supervisors. Data recorded during supervision visits are dependent on the capacity of the supervisor, requiring standardization of supervisors’ knowledge and skills. Given that many of the indicators reflect not only the performance of CHWs, but also the performance of CHW supervisors and the health system in general, supervisors may be inclined to report more favourable results than are in fact the case, which could further distort indicator measurements. For most countries, reporting indicators via supervision checklists would require a reform of supervision systems and how data are managed, which may not be realistic. In addition to these changes, additional work would likely be needed at health facility and district level to compute indicators that require data from multiple sources. Ministries of health would need to consider carefully the additional burden that this might place on already overworked staff and chose only those indicators that add value to programme management.

Next steps for the indicator guide and iCCM monitoring

This review, while of limited scope, offers insight for how to improve the Indicator Guide and further develop robust M&E systems. We suggest that some of the routine monitoring indicators, in their current form, may be overly difficult to measure and therefore need revising. The indicators concerning supply chain management and performance quality assurance are particularly problematic and require re-thinking: the data needed for the current indicators on these topics are not collected or aggregated through existing tools, and the countries that do collect these data rely on supervision checklists to do so. Some indicators, such as 7.6 ‘correct case management’ and 7.8 ‘complete and consistent registration’, require complex data collection or aggregation and should be rethought entirely or collected only via special studies. Other indicators are more promising, such as those concerning service delivery and human resources. If countries can collect up-to-date CHW deployment data on a routine basis, these indicators should be straightforward to measure without adding to the reporting burden on CHWs and CHW supervisors.

We also suggest that the overall list of routine monitoring indicators should be shortened. Monitoring iCCM nationally and globally requires standardization to produce comparable data across countries. Implementing a core set of standardized indicators will be more effective, and more feasible, if countries have fewer indicators to measure: for example, three to five high-value routine monitoring indicators. Criteria for these indicators could include (1) effort required for data collection, aggregation and computation; (2) reliability of measurement and interpretation; and (3) utility for both for global and national/sub-national stakeholders and for health workers and CHWs at the point of service. A smaller set of indicators would also guard against collecting information that does not strengthen programme implementation. Indicator choices made at the national and global level have real-world consequences for CHWs and the time CHWs spend on administrative paperwork at the expense of other activities. Ensuring that CHWs and CHW supervisors use their time as efficiently and fruitfully as possible should be a high priority for everyone involved in monitoring iCCM. (The findings from this article, along with the above suggestions, have been presented to the M&E subgroup of the global iCCM Task Force.)

Finally, we make an assumption in this article that CHWs and health workers are using the M&E tools as intended. Clearly this may not be the case in some settings (Guenther et al. 2014). Further analysis about how M&E tools are being used and data quality would add further insight into the practicability of the indicators and how they could best be revised.

Implications for ministries of health and implementing partners

Ultimately it is for countries to determine which indicators they will monitor through routine sources. Ministries of health and implementing partners face many choices in establishing iCCM monitoring systems: which information to collect; what indicators to report and at what level of the health system; which tools to use; how to aggregate, communicate and compile data; and, ultimately, how to use the data to improve the iCCM programme and save lives. Decisions on which indicators to measure should reflect a country’s resources and context, including the capacity of CHWs to complete reporting forms, the workload of staff at health facilities and integration for reporting with other health programmes. Indicator choices should also reflect an understanding of how M&E data will be used at different levels of the health system. Reporting requires resources, and collecting data that are not used is a misuse of resources. These decisions should be considered by countries over time and articulated in national iCCM M&E policies.

NGOs and other agencies should provide technical assistance to countries on choosing indicators, revising tools and protocols, and developing data analysis strategies. Donors should continue to provide financial support for monitoring iCCM. As a country’s iCCM programme grows and develops, international actors could support the appropriate introduction of new technology for data collection (such as mobile phones and computerized data analysis), quality assurance and reporting through an integrated HMIS. International agencies should also consider the sustainability of M&E systems. Donors and other implementing partners have a role to play in resisting duplicate systems and advocating wherever possible for routine monitoring that is government led and integrated with other routine monitoring systems. Many countries are now using District Health Information System (DHIS)-2 as their routine data management system, which is capable of incorporating iCCM data (DHIS-2 2015). Finally, all parties should promote not only the collection of M&E data, but also its use. The burden that M&E systems place on CHWs, health facility workers and other health staff is not insignificant. Global actors cannot continue to advocate the expansion of M&E systems without complementary support for data use. To establish effective and efficient M&E systems, it is critical that we determine which information is genuinely needed at which level of the health system, and design M&E systems with those actors and purposes in mind. If M&E data are useful for researchers and global actors, but not for those on the ground, the continued promotion of those indicators is unlikely to translate to improve programming and health outcomes (WHO 2015).

Improving national and global monitoring of iCCM programmes will take time and learning from in-country experiences. Countries are reviewing the Indicator Guide, considering its implications and adopting some of the indicators to monitor their programmes. In the meantime, the iCCM Task Force M&E subgroup is in the process of mapping countries that have adopted the DHIS and how many have added iCCM indicators to the community platform. Using this information, the CCM TF should consult ministries of health and implementing partners to develop a revised set of recommended indicators, to drive forward the dialogue on M&E for iCCM and establish a focus on rationalizing data collection, improving data quality and use for decision making. The revised indicators should respond to the challenges identified in this article, in-country experiences and other related research.

Limitations

Our analysis was limited to the template monitoring tools provided to us by ministries of health and implementing partners. We made every effort to ensure that the tools we reviewed were the most current and complete versions of the tools at the time of writing; by contacting or meeting with key informants to discuss the tools, and by sending a final version of this analysis for their approval. If other monitoring tools are being used in a country, the feasibility of calculating routine monitoring indicators in that country may be different to what is presented here. In any case, the sample of countries and tools reviewed in this study was not meant to be representative of all countries, but rather a purposive sample to generate initial lessons for improving the Indicator Guide.

This article discusses the feasibility of collecting the Indicator Guide’s routine monitoring indicators, assuming that the iCCM M&E systems in the 10 countries are working as expected. Our analysis did not address issues of data quality, data completeness or data use. These issues certainly need attention and study, but they were not the focus of this study. A related study that addresses data quality and data use would likely have additional findings on the relevance and practicability of the Indicator Guide. We hope that this article will be a precursor to further research, both on the Indicator Guide itself and on the broader challenges and opportunities for monitoring large-scale iCCM programmes.

Conclusions

Developing robust national M&E systems for iCCM will take time and learning from in-country experiences. Some of the routine monitoring indicators currently recommended by the iCCM Task Force need revising, and the total list of recommended indicators should be shortened. Routine monitoring of iCCM will be more feasible, effective and efficient if countries are encouraged to measure a smaller set of high-value indicators that are easy to implement, reliably interpreted and useful for both global stakeholders and frontline health workers themselves. We advocate further research on iCCM indicators and on the broader challenges and opportunities for monitoring large-scale iCCM programmes.

Supplementary data

Supplementary data are available at HEAPOL online