‘Do you trust those data?’—a mixed-methods study assessing the quality of data reported by community health workers in Kenya and Malawi

Abstract

High-quality data are essential to monitor and evaluate community health worker (CHW) programmes in low- and middle-income countries striving towards universal health coverage. This mixed-methods study was conducted in two purposively selected districts in Kenya (where volunteers collect data) and two in Malawi (where health surveillance assistants are a paid cadre). We calculated data verification ratios to quantify reporting consistency for selected health indicators over 3 months across 339 registers and 72 summary reports. These indicators are related to antenatal care, skilled delivery, immunization, growth monitoring and nutrition in Kenya; new cases, danger signs, drug stock-outs and under-five mortality in Malawi. We used qualitative methods to explore perceptions of data quality with 52 CHWs in Kenya, 83 CHWs in Malawi and 36 key informants. We analysed these data using a framework approach assisted by NVivo11. We found that only 15% of data were reported consistently between CHWs and their supervisors in both contexts. We found remarkable similarities in our qualitative data in Kenya and Malawi. Barriers to data quality mirrored those previously reported elsewhere including unavailability of data collection and reporting tools; inadequate training and supervision; lack of quality control mechanisms; and inadequate register completion. In addition, we found that CHWs experienced tensions at the interface between the formal health system and the communities they served, mediated by the social and cultural expectations of their role. These issues affected data quality in both contexts with reports of difficulties in negotiating gender norms leading to skipping sensitive questions when completing registers; fabrication of data; lack of trust in the data; and limited use of data for decision-making. While routine systems need strengthening, these more nuanced issues also need addressing. This is backed up by our finding of the high value placed on supportive supervision as an enabler of data quality.

Key Messages

• High-quality data are essential to monitor and evaluate the performance, quality, coverage and equity of community health programmes.

• Community-level health data are not used routinely for decision-making because of lack of trust in their quality.

• Data Quality Assessments provide a feasible method of regularly assessing data quality and identifying gaps that can be addressed to strengthen community health systems.

• Supportive supervision should be provided to community health workers to ensure they collect and report high-quality data. This should include consideration of their unique interface role between the formal health system and the communities they serve, mediated by the social and cultural expectations of their role that may create barriers to data quality, e.g. in relation to gender norms.

Introduction

Community-level health data are critical to understanding progress towards universal health coverage (UHC) and achieving sustainable development goals. High-quality data underpin problem identification and quality improvement at the community and facility levels and are used for evidence-based priority setting, resource allocation and health strategy development at the district and national levels (Mitsunaga et al., 2015; Wagenaar et al., 2016; McCollum et al., 2016a). Data collected at the household level, through nationally representative samples for the purposes of demographic and health surveillance, contribute to understanding of equity, health status and determinants at the community level (Hedley and Mcghee, 2004; Hotchkiss et al., 2012). However, large-scale surveys are infrequent, costly and do not reflect the full scope of community health programmes (Bryce et al., 2013). Routine community health programme data have the potential to fill this gap by assessing the performance, quality, coverage and equity of community health worker (CHW) programmes (Hedley and Mcghee, 2004; Lewin et al., 2010; Gilmore and Mcauliffe, 2013; Perry et al., 2014b). In national community health strategies, data collection and reporting are included amongst the tasks expected of CHWs, and traditionally CHWs record data in paper-based tools that are collated at higher levels of the health system and are often linked to facility-based and national health information systems (Braun et al., 2013).

In practice, the quality of community-level health data reported by CHWs in low- and middle-income countries (LMIC) is often low. Studies in Ghana, Kenya, Malawi, Pakistan and Rwanda have found that under- and over-reporting by CHWs are common (Helleringer et al., 2010; Mahmood and Ayub, 2010; Otieno et al., 2012; Admon et al., 2013; Mitsunaga et al., 2013; Otieno-Odawa and Kaseje, 2014; Yourkavitch et al., 2016). Low-quality data have resulted in little demand for and use of community health information systems in decision-making (Wagenaar et al., 2016). Factors that have been found to contribute to low-quality community-level health data include inadequate and incompatible data collection and reporting tools, lack of training of CHWs on data management, high workload among insufficient numbers of supervisors for CHWs, and the perception of CHWs that the data they report is not used, thereby reducing motivation to stringently manage data (Ekirapa et al., 2013; Yourkavitch et al., 2016).

Both Kenya and Malawi have recently revised their community health strategies with an emphasis on improved data quality, although neither provide a definition for high-quality data, describe how data quality should be assessed or give guidance on data quality improvement (Kenya Ministry of Health Community Health Unit, 2014; Malawi Ministry of Health, 2017). Assessing and improving data quality are essential in building policymaker and community trust in the work that community health programmes report. Such assessments underpin data use for priority setting, problem identification, tracking progress and measuring improvement. We set out to conduct a data quality assessment and explore barriers and enablers to data quality in two African contexts large-scale community health programmes. Figure 1 summarizes and compares the reporting (levels, aggregation points and data flow) in the community health information systems of Kenya and Malawi, while Table 1 adds information on responsibilities for and frequency of data collection in each system.

Figure 1.

Reporting levels and data flow of community health information systems in Kenya and Malawi.

Table 1.

Data collection and reporting tools

Country	Data collection and reporting tool	Data content/data source	How data are recorded	Completed by	Frequency of data collection
Kenya	MOH 514 Service Delivery Log Book (paper-based)	Indicators regarding: maternal health status and services; newborn and child health status and services; referrals; defaulters; death—these data are obtained during household visits	✔, × or N/A are used to indicate a positive case/action carried out, negative case/action not carried out or not applicable, respectively	CHV	Continuous (completed during household visits)
	MOH 515 CHEW Summary (paper-based)	Aggregated total of the data reported by the CHVs of each Community Unit in MOH 514 Service Delivery Log Books	Numerical data—totals are recorded for each indicator	CHEW	Monthly
Malawi	Village Clinic Register (paper-based)	Indicators regarding under-fives: new cases; referrals with danger signs; referrals made because of drug stock-out; deaths within 7 days of receiving treatment at a village clinic; medical supplies—these data are obtained during village clinic visits	Numerical data—totals are recorded for each indicator	HSA	Continuous (completed during village clinic visits)
	Form 1A	Aggregated total of the data reported for a catchment area in Village Clinic Register	Numerical data—totals are recorded for each indicator	HSA	Monthly
	Village Clinics Monthly Report Form for Under Fives (paper-based)
	Form 1B	Aggregated total of the data reported by the HSAs attached to a primary healthcare facility in Village Clinics Monthly Report Forms for Under Fives (Form 1A)	Numerical data—totals are recorded for each indicator	SHSA	Monthly
	Village Clinics Monthly Consolidated Report—Health facility level (paper-based)

In Kenya, there are two main cadres at the community level: community health volunteers (CHVs) and community health extension workers (CHEWs). CHVs are trained to provide promotive, preventive and simple curative health services at the household level. CHEWs are recruited and trained by the Ministry of Health and their main role is training and supervising CHVs (Kenya Ministry of Health Community Health Unit, 2014). The delivery of community health services is centred around community units, comprised of ∼5000 persons and linked to one primary healthcare facility. Each community unit should be served by five CHEWs and 10 CHVs (Mwitari et al., 2016). CHVs should record household visits and activities in a paper-based Ministry of Health (MOH) 514 Service Delivery Log Book (MOH 514) and report monthly to the CHEW, who aggregates their data to complete the paper-based MOH 515 CHEW Summary form (MOH 515) that is in turn submitted to the sub-county health management office for digital entry into the web-based national health information system, Kenya Health Information System Aggregate.

In Malawi, there are CHVs, health surveillance assistants (HSAs) and senior health surveillance assistants (SHSAs). The two latter cadres are recruited and trained by the Ministry of Health. CHVs assist HSAs to provide the community components of Malawi’s Essential Health Package; HSAs supervise CHVs; and SHSAs supervise HSAs. The delivery of community health services in Malawi is centred around catchment areas linked to one primary healthcare facility. Within these, there should be one HSA per 1000 persons and one SHSA per 10 HSAs, performing supervision activities across ∼10 catchment areas (Malawi Ministry of Health, 2017). HSAs complete a paper-based Village Clinic Register with assistance from CHVs. They are also responsible for collation of Village Clinic Registers to complete a paper-based Village Clinics Monthly Report Form for Under Fives (Form 1A) for submission to their SHSA, who aggregates these in a paper-based Form 1B submitted to the district health office for digital entry into the web-based national health information system, DHIS2.

Methods

We used mixed methods for our study similar to those that have been used for data quality assessments of health information systems in other sub-Saharan African countries (Ledikwe et al., 2014; Yourkavitch et al., 2016). Quantitative data verification ratios enabled us to determine the magnitude of the data quality problems and the types of errors. Qualitative methods were used to explore how community-level health data management worked in practice, the enablers and barriers to data quality and the use of community-level health data in decision-making.

Study sites

Our study was nested within a larger multi-country study (REACHOUT) investigating the equity, effectiveness and efficiency of close-to-community provision of health care in six countries in Africa and Asia (REACHOUT Consortium, 2019). The study was conducted in four REACHOUT districts, two in each country, where baseline context analyses had revealed lack of trust in community health data systems (Mireku et al., 2014; Nyirenda et al., 2014). In Kenya, we purposively selected two rural community units in Kitui County (Museve and Township) and two urban community units (Bangladesh and Maili Saba) in Nairobi County. In Malawi, we purposively selected the rural catchment areas linked to Chioshya and Kochilira health centres in Mchinji District and Chinguluwe and Lifuwu health centres in Salima District. All tools completed by each CHW in our selected study sites were included in the study.

Data collection

We adapted MEASURE Evaluation’s Multi Indicator Routine Data Quality Assessment (M-RDQA) Tool to create tools relevant to Kenya’s and Malawi’s community health information systems. These tools were derived from a conceptual framework adapted from MEASURE Evaluation’s Data Quality Assessment conceptual framework (Figure 2). The M-RDQA Tool is a Microsoft Excel workbook that contains two parts for data collection: verification of reported data for selected indicators and health information system assessment.

Figure 2.

Community health information system data quality assessment conceptual framework adapted from the MEASURE Evaluation’s Data Quality Assessment conceptual framework (MEASURE Evaluation, 2015).

Quantitative verification of reported data for selected indicators

We selected up to a maximum of seven indicators for verification that aligned with the community health programme priorities of maternal and child health in Kenya and child health in Malawi (Table 2). The reporting periods assessed were March to May 2016 in Kenya and May to July 2017 in Malawi.

Table 2.

Indicators selected for calculation of data verification ratios

	Kenya	Malawi
1.	Pregnant woman referred for antenatal care	Children 2–59 months with new cases (of fever, malaria, diarrhoea, fast breathing, pneumonia and others including red eye)
2.	Pregnant woman referred for skilled delivery	Children 2–59 months with referral made due to danger signs
3.	Maternal delivery accompanied by skilled birth attendant	Children 2–59 months with referral made due to drug stock-out
4.	Child 0–11 months referred for immunization	Children 2–59 months that died within 7 days of receiving treatment at the village clinic
5.	Child 0–59 months participating in growth monitoring
6.	Child 6–59 months with mid-upper arm circumference (red) indicating severe malnutrition
7.	Child 6–59 months with mid-upper arm circumference (yellow) indicating moderate malnutrition

We manually added up (re-aggregated) the raw data available from the lowest level CHWs for each selected indicator for each month and entered these totals into the M-RDQA tool. Next, we entered values reported for the same indicators in supervisors’ summary forms for the same reporting period. Data verification ratios were calculated by dividing the re-aggregated total of data recorded by the lowest level CHWs by the value reported by their supervisors at the next level. A data verification ratio of 1 indicated complete consistency between the two levels of reporting. Data verification ratios >1 and <1 indicated a higher or lower number reported by CHVs/HSAs than by their supervisors, respectively.

Qualitative health information system assessment

We purposively selected participants based on the roles that correspond to the reporting levels shown in Figures 1 and 2. We used focus group discussions (FGDs) with CHVs as these allow exploration of opinions and experiences that would be less forthcoming in a one-to-one interview (Kitzinger, 1995). This is useful because group discussion provided a forum where gaps could be revealed with empathy from peers, rather than engendering a sense of judgement about individual performance. We conducted 13 mixed-sex FGDs (with 52 CHVs in Kenya and 83 CHVs in Malawi).

Key informants were selected based on the perspective they might have due to their working relationship with CHWs and/or their contact with community level health data (see Table 3). We used semi-structured interviews with the supervisors of CHVs and key informants. These combined questions on issues the interviewer would like to explore and permitted deviation to follow an idea through or gain further detail (Gill et al., 2008). A total of 36 interviews were conducted across the two countries until saturation was reached.

Table 3.

Study participants

Type of participant	Total
Kenya
Community level
CHVs	54
CHEW	4
Health facility level
Facility in-charge	4
Sub-county/district level
Sub-county health record information officers	3
Sub-county community health strategy focal persons	3
Malawi
Community level
CHVs	70
HSAs	13
SHSAs
Health facility level
Facility in-charge	14
Sub-county/district level
Data clerks	12

Questions focused on the functional components of a community health information system (see Figure 2). We explored the understanding of what data were collected and by whom, how these data were recorded, who were responsible for aggregation of data, existence of quality control mechanisms, how data flowed up the system, who entered these data and where, and to whom data were reported. See Supplementary Appendix SA1 for the questions that comprise the health information system assessment section of the M-RDQA tool. These questions directly informed the topic guides used in Kenya and Malawi. We also asked about feedback and supervision processes and perceived enablers and barriers to data quality. Topic guides were translated into Kiswahili and Chichewa in Kenya and Malawi, respectively, for use with CHVs and piloted before use. For all other participants, English was used. Interviews were conducted by the authors RNR and KC, as well as trained research assistants. Data were digitally recorded and transcribed into Microsoft Word documents for inter-country analysis. In the case of FGDs with CHVs, these were translated into English at the time of transcription. Transcripts were checked against the audio files.

All data collectors participated in qualitative data analysis, as did other experienced qualitative researchers. At first, we established a coding framework deductively using the topic guides and MEASURE Evaluation’s (2015) dimensions of data quality. We then worked iteratively and adjusted this coding framework inductively from additional, unexpected or divergent data. We used NVivo Version 11 to code transcripts and assist with the analysis, which used a thematic framework approach. Emerging themes were grouped and preliminary findings presented to the Ministry of Health and other stakeholders in both countries to corroborate our findings and finalize the narrative per theme.

Results

The selected sites in both Kenya and Malawi had evidence of significant under- and over-reporting of community health data. Kitui County in Kenya was an outlier in that we found there were no data reported for the study sites therein for at least 18 months prior to our study. This was attributed to total unavailability of the necessary MOH data collection and reporting tools. When exploring the enablers and barriers to data quality, we found remarkable similarities between the two countries and districts even though the typology of CHWs varied (with Kenyan CHVs being volunteers), the extent of devolution and decentralization, the health systems and cultural contexts were all different. In this section, we present the quantitative results first and then the themes that emerged in the five areas related to our conceptual framework for community health information system assessment: monitoring and evaluation structure, functions and capabilities; indicators and guidelines; data collection and reporting tools; data management processes; and links with the national reporting system.

Verification of reported data for selected indicators

In Kenya, the data verification exercise was done using 46 service delivery log books and six CHEW summaries from Maili Saba and Bangladesh community units in Nairobi County. It was not possible to calculate data verification ratios for the Kitui sites as there was no reporting for at least 18 months prior to our data collection. In Malawi, the exercise was done using 293 Form 1As and 66 Form 1Bs. Only 15% of the data reported in the 3-month period in 2016 were consistent across levels. Tables 4 and 5 show inconsistency in most values reported by CHWs and their supervisors, with data verification ratios ranging from 0.00 to 10.00. While there appeared to be more under-reporting in Malawi than in Kenya (where there was a tendency to over-reporting), we were unable to determine any pattern across sites, individual CHWs or months in either country. No site consistently collated data. Our qualitative findings are set out in more detail below and reveal multiple reasons for the poor match in verification ratios.

Table 4.

Data verification ratios for Kenya

Table 5.

Data verification ratios for Malawi

Health information system assessment

Monitoring and evaluation structure, functions and capabilities at community level

In Kenya, all participants indicated that CHVs are responsible for reporting monthly on a standard set of indicators using MOH 514 CHV Service Delivery Logbook. In Malawi, HSAs were recognized as responsible for reporting community-level health data but often relied on CHVs to collect these data. Many participants reported that CHVs in Malawi collected data on an ad hoc basis when data were required for a service, relying on HSAs to tell them what data were required when. There were, however, no standard data collection and reporting tools produced by the Government of Malawi for use by CHVs, who reported recording demographic data; immunization data; water, hygiene and sanitation data on loose papers, in notebooks or even on cardboard boxes (see Links with National Reporting Systems section below for more details). They then provided these data to HSAs to complete village clinic registers.

In both countries, all participants stated that they were aware of the role of CHEWs and HSAs, respectively, to aggregate and complete a single summary form of community-level health data for their community unit or catchment area. However, CHEWs in Kenya did not routinely verify the data they received from CHVs, apart from instances such as verifying reports of maternal and child deaths. In Malawi on the other hand, many participants, including CHVs, reported that HSAs conducted occasional household visits to verify data reported by CHVs.

Lack of training on data management emerged as a key theme and was linked to an overall lack of training to provide the key services delivered at the community level. CHWs in both Kenya and Malawi reported that in most cases, a CHV had either received no training at all or had only received training limited to certain health service areas. For example, in Kitui, Kenya, CHVs had been trained in water, sanitation and hygiene but not in maternal and child health. In both countries, older CHVs reported no refreshers since their initial training years previously. All CHVs reported they had not received formal training on data collection and reporting, as illustrated by this typical quote:

We are supposed to be trained how to fill that so that when you ask a person and he/she tells you, you get to know how to record it. They have put X or tick or N/A — we don’t understand what should be put in tick or X or N/A (CHV, Nairobi, Kenya).

In Kenya it emerged that the gender of CHVs can have an impact on their ability to collect all the data that they should. For example, several male CHV respondents reported feeling that it is not culturally appropriate to discuss sexual and reproductive health with women. Female CHVs also reported that family planning is not a topic that can be discussed openly in households between two women due to some men’s disapproval. Similar findings were not elicited in Malawi, where this question was not probed by the researchers.

Indicator definitions and reporting guidelines

In Kenya, the CHV Service Delivery Logbook contained instructions on data capture. However, it was found that in Kitui, none of the CHVs had been trained on these indicator definitions, and in Nairobi some but not all had received orientation. CHEWs reported varying levels of confidence that CHVs knew what to enter. In both Kenya and Malawi, CHVs often did not submit their data on time due to irregular meetings with their supervisors, unavailability of data collection and reporting tools and competing priorities. Only CHVs in Nairobi knew exactly when and to whom they should submit their data. In Kitui, CHVs did not know of a specific date for submission and did not even know to whom they should submit their data, as this quote illustrates:

We don’t see the CHEW and from last year we haven’t filled in anything. We should see the CHEW but some of us don’t even know him and decide on how we will be doing work. This thing is important, but you can write and end up just walking not knowing where to take it (CHV, Kitui, Kenya).

In Malawi, CHVs reported that they collect the data they have been instructed to by the HSA or the data that they believe are expected from them. All SHSAs reported confidence that the CHVs understood what data they were recording because CHVs worked closely with HSAs, even occasionally conducting household visits together. CHVs reported to HSAs on an intermittent basis, with differing timelines dependent on the health service area or programme for which data are being collected.

Data collection and reporting tools

In Kenya, participants reported that CHVs did not use CHV service delivery logbooks during household visits. In Kitui, this was attributed to unavailability of these tools. In Nairobi, this was attributed to difficulty in carrying the registers around due to their large size and to the fact that the tool is branded with Kenya’s Coat of Arms, fostering the false perception that CHVs are paid, as this quote illustrates:

Going with this book there [on household visits] is cumbersome and risky and people assume that we are paid (CHV, Nairobi, Kenya).

Many CHVs criticized the fact that the CHV service delivery logbook was only available in English and contained technical language, reporting that some CHVs sought out other CHVs to interpret for them.

In Malawi, questions regarding the design and use of data collection and reporting tools for CHVs were not applicable because such tools did not exist; CHVs reported that they all record data in different ways as this quote illustrates:

…there isn’t any expertise given in how to perform our work; every work must have guidelines to follow, but when we don’t have such then we just do as we see fit just to get the work done (CHV, Salima, Malawi).

However, several CHVs reported that when they recorded data, they did try to record age and sex, particularly when collecting data pertaining to children.

Data management processes

Our participants indicated that CHVs stored the data they collect in their homes with supervisors citing a lack of adequate storage space leading to data forms being lost and potential breaches of confidentiality. Neither country had a formal quality control mechanism in place for data reporting, nor participants reported both over- and under-reporting as a result. In Kenya, over-reporting was thought to occur due to supervisors obtaining data from a source other than the community data (such as a facility register) and using this in their summary. Some supervisors had developed informal mechanisms for quality control. For example, in Malawi, participants reported that HSAs conducted household visits, as illustrated by the following quote:

Last year [2016] between November and December we were issuing out mosquito nets—we gave this task to volunteers to collect data. I was asked by my boss to go and cross check—I just took a form and visited and asked houses in the village without involving volunteers and the data was matching. That gave me a picture that they were collecting true information (SHSA, Salima, Malawi).

In Maili Saba, Kenya, the CHEW met with all CHVs with their service delivery logbooks on an appointed day to lead collation into the CHEW summary as a group activity during which any queries or discrepancies were corrected:

Really during that compilation that we are doing that’s the time we also try to check that whatever data I’m writing is truly a true reflection and if there is any issue with a particular CHV there, we are able to discuss the true reflection of what has happened (CHEW, Nairobi, Kenya).

In both countries, when supervisors of CHVs and key informants were asked about how data were verified, they reported an inability to track community referrals due to poor linkage with primary healthcare facilities. In Kenya, there was an official referral form, but this tool was unavailable in the community units assessed. In Malawi, there was no such official referral tool. There were no specific systems for recording referrals received from the community at the facility level in either country.

Poor supervision was also a common problem. All participants in both Kenya and Malawi reported that supervisors of CHVs did not make copies of the data they receive from CHVs. Secondly, CHVs reported lack of feedback and supportive supervision on the quality of community level health data as illustrated by the following quote:

They should be telling us if we have done a good job or not and give us information because we need feedback — in that way we will be encouraged that we know the work. With what is happening it is as if we do not make mistakes (CHV, Salima, Malawi).

Despite these informal systems, asking ‘do you trust community level health data?’ revealed concerns regarding fabrication of data among CHWs, with terms like ‘cooking’ data recurring in the transcripts from both countries. Two CHEWs in Kenya openly admitted to fabricating data themselves:

That happens because we are forced to bring the 515s, to bring data at the end of the month and you do not have the raw data to compile that…if they are not there and you are told to submit a report, you have to cook (CHEW, Kitui, Kenya).

Similarly in Malawi:

…what sometimes happens is that HSAs do not update their data and so if the data is needed abruptly, they will just fabricate a new set of data (SHSA, Mchinji, Malawi).

Links with national reporting systems

We found that data collection and reporting tools were often unavailable, having not been printed or distributed. The limited availability of standard data collection and reporting tools resulted in improvization. Supervisors took it upon themselves to photocopy tools in Kenya, or, as reported in Malawi, HSAs create their own tools:

We currently improvise. We use whatever paper we come across and we design it based on the original questionnaire’s format (SHSA, Mchinji, Malawi).

Exercise books/plain papers were used to record data, and these were either purchased at the health workers’ own expense or torn from their children’s school exercise books. In some instances in Malawi, cardboard boxes were torn up for recording data.

In both countries, CHVs did not know the flow of data after reporting to their supervisors. Parallel reporting to specific NGOs and/or vertical health programmes was said to be common, especially in Malawi. It was reported that in such scenarios CHVs were provided with multiple data collection and reporting tools, creating extra workload.

Finally, participants were asked if they trusted community-level health data. The almost unanimous response was only when it had been verified by supervisors. The most common reason for lack of trust in community-level health was that CHVs simply did not document all their activities:

Any work minus documentation, it’s no work done. So that’s where we have a lot of problems — documentation. They provide nice services in the communities, but they don’t document. How do you trust that one ? (Data Clerk, Mchinji, Malawi).

In Kenya, this question also revealed that CHVs only visited a small proportion of the households that they were expected to each month. This meant that even if all the other dimensions of data quality were met, community-level health data would not be high quality due to incompleteness and its inability to give a picture of coverage to higher levels of the system:

These CHVs do not go to all the households but whatever they give is truly a reflection of those households they’ve visited; but this is only a fraction of the entire number of households that are supposed to be served (CHEW, Nairobi, Kenya).

Discussion

Our mixed-methods assessment of community health data found large discrepancies in selected community units in both Kenya and Malawi between the values reported by CHVs and those reported by their supervisors. Overall only 15% of the data reported were consistent during the 3-month period. Participants at various levels of the health information system raised concerns about the quality of community-level health data and its limited potential to contribute to planning. We found multiple similarities underlying causes of this poor data quality in the two countries even though one has HSAs paid and integrated into the health system (Malawi) and one used volunteers (Kenya). Both reported unavailability of standard data collection and reporting tools and limited training for CHVs on maternal and child health issues as well as on data entry. This problem was exacerbated by parallel reporting requirements of vertical programmes, resulting in multiple tools and duplication of efforts that was particularly prominent in the Malawian context. In-depth exploration revealed that CHWs experience tensions at the interface between the formal health system and the community they serve and are affected by the social and cultural expectations of their role with reports of fabrication of data in both countries and evidence that CHW’s gender can affect reporting. The value of a supportive approach to supervision as an enabler of data quality was stressed by community participants in both contexts, with examples of what this term meant to them in practice.

Our findings are in keeping with descriptions of the low quality of community level health data in other sites in Kenya and Malawi as well as other LMICs (Naimoli et al., 2015; Frontline Health Workers Coalition, 2016). We found more under-reporting of community data in Malawi than a similar exercise in four districts in central and northern Malawi a few years before our study (Yourkavitch et al., 2016) and think this under-reporting was due to the fact that our districts were not receiving support to integrated community case management through the World Health Organization’s Rapid Access Expansion projects. This study also conducted qualitative work that mirrored the need for supportive supervision we describe, although fabrication of data and the importance of gender and cultural sensitivities did not emerge.

CHWs are often burdened by clashing vertical programmes that fail to consider the wider workload and fail to integrate reporting and supervision structures (Baatiema et al., 2016; McCollum et al., 2016b; MEASURE Evaluation PIMA, 2016; Seutloali et al., 2018). This results in CHWs prioritizing the reporting of certain data at the expense of other data, based on factors such as remuneration (Scott et al., 2018). The quality of community health could be strengthened through paying adequate attention to the co-ordination and comprehensive planning of community health programmes (Mireku et al., 2014; Nyirenda et al., 2014; Kok et al., 2015a,b).

High-quality community-level health data are an essential first step in improving the quality of care at the community level. The launch in 2018 of three key global documents has resulted in an increased focus on the need for quality in healthcare in LMICs, but these documents largely fail to address quality issues in community health (Kruk et al., 2018; National Academies of Sciences, Engineering, and Medicine, 2018; World Health Organization et al., 2018). There are, however, a growing number of examples of how high-quality data can underpin quality improvement efforts for community health services, with resultant impacts on health outcomes. In Ethiopia, Kenya, South Africa and Tanzania, quality improvement teams have successfully used local data to identify gaps in community health services and develop practical solutions (Lunsford et al., 2015; Miller et al., 2016; Horwood et al., 2017; Otiso et al., 2018). In Malawi, training on data analysis and use has resulted in greater ownership by CHWs and facility health workers and the use of data for quality improvement (Hazel et al., 2017), and in Rwanda and Zambia, the embedding of data quality assessments has had the same effect (Gimbel et al., 2017). The WHO guidelines on health policy and system support to optimize CHW programmes stress the importance of data quality (World Health Organization, 2018), but in practice, widespread mistrust of community health data by decision-makers means its potential to bring about quality improvement will not be realized.

Training, supervision and the availability of registers are all required in order to strengthen community health information systems. In Ethiopia, Kenya, Malawi and Mali, it has been shown that with training targeted towards specific tasks, CHWs can collect accurate data (Mwangome et al., 2012; Silva et al., 2016). In certain sites in Malawi, changing the language of data collection and reporting tools used by CHWs from English to the local language reduced the number of errors in data collection (Admon et al., 2013). It is during supervision meetings between CHWs and their supervisors that paper-based data reporting tools are submitted. Without such meetings, CHWs feel that there is no demand for their data and feel unsupported and demotivated. The desire of CHVs for more supervision and feedback is a common finding across CHW programmes (Perry et al., 2014a; Ndima et al., 2015; Henry et al., 2016; Phiri et al., 2017). Systematic reviews on the performance of CHWs have found that frequent supervision with supportive approaches and a focus on quality assurance/problem-solving, as well as continuous training, are effective ways to improve CHV performance (Hill et al., 2014; Kok et al., 2015a). Local supervisors can best understand the context in which CHVs work, creating an immediate opportunity for problem-solving and improved data quality. For example, when gender norms, roles and relations affect the ability of male CHVs to ask and report questions concerning pregnancy and family planning (Steege et al., 2018).

Our study found paper-based records in place in all sites but it is not clear how long this system will remain in place. Mobile health (mHealth) is playing an increasingly important role in data capture for community health services in LMICs and has been shown to positively impact data timeliness, completeness and accuracy (Braun et al., 2013; Kallander et al., 2013; Agarwal et al., 2015; Joos et al., 2016; Atnafu et al., 2017). In addition, when combined with the use of global position system technology common to smartphones, mHealth also has the potential to give a picture of programme coverage (Kazi et al., 2017). mHealth alone, however, is not a panacea to community health data capture, as data capture is only the first step in a chain of data use, may be expensive and unsustainable and poses technical, ethical and legislative challenges, including loss of ownership of data and confidentiality (Wambugu and Villella, 2016).

Limitations

There were several limitations of our study. We selected maternal and child health indicators that were county or district priorities and feel other indicators would have yielded similar results, although it is possible that we would have seen worse data quality due to limited attention, feedback or follow-up from county or district health management. The numbers reported for some of the indicators assessed were low. For example, severe malnutrition was a rare event in a small community unit and only a handful of cases were identified each month. This led to a disproportionate impact on data verifications ratios. We did not assess the accuracy of what was recorded during household visits through direct observation and this may have introduced data errors. We conducted the quantitative part of the study after qualitative data collection and thus missed opportunities to inform the development of our topic guides in probing areas of concern arising from the data quality assessments. In Malawi, due to unstructured and informal routine data collection amongst CHWs, it was not possible to obtain the village clinic registers to see how they compared with data reported in Form 1A. Lastly, the study did not assess the quality of reporting by personal characteristics of CHWs (e.g. age, gender, living in urban area/living in rural area, level of education) to determine associations with data quality. Qualitative data were only collected from a small number of sites compared with the total number of sites in each country, meaning our findings may not be totally representative.

Conclusion and recommendations

The quality of community health data is an essential component of tracking progress towards UHC, but current systems and structures are unable to ensure this cornerstone is in place. Our findings underpin a number of policy and practice recommendations for the strengthening of community health information systems in LMICs and are presented in Box 1 As countries develop, revise or scale-up their community health programmes, it is imperative that data collection, reporting and quality, are addressed holistically in the context of supportive supervision, so that reliable, trusted information can be generated for use in decision-making at all levels of the health system.

Box 1 Recommendations for strengthening of community health information systems

• CHVs should have official, standardized data collection and reporting tools (including referral tools) that are designed with CHVs in mind, e.g. the language used, simplicity, size, etc.

• Governments, donors and partners should co-ordinate reporting efforts.

• Training on data management should be part of standard training for CHVs and their supervisors.

• Primary healthcare facilities should store and track referrals received from CHVs.

• Regular supervision meetings between CHVs and their supervisors should include data feedback and data quality as a standing agenda item.

• Supportive supervision should incorporate discussion of the data findings to address difficulties in data entry and highlight to CHVs that there is demand for the data that they report.

• Regular data quality assessments and data verification exercises should be carried out to monitor and identify gaps in data quality.

• High-quality data should be analysed and used to identify problems, develop solutions and improve programme quality at all levels, increasing the confidence of decision-makers.