Study on external consistency of health management information system data at district level across India

ABSTRACT

Background:

The WHO has provided toolkit for data quality review of Health Management Information System (HMIS) data, with external consistency of coverage rate being one dimension.

Objective:

To assess the level of external consistency of HMIS data compared with National Family Health Survey-5 (NFHS-5) data at district level across India.

Methods:

We used secondary data on health service delivery indicators across districts at meso-level collected from HMIS and NFHS-5 website. We collected data on 7 indicators: Mothers registered within trimester I, mothers with at least 4 antenatal visits, mothers last birth protected against tetanus, institutional births, births delivered by Caesarean section, sex ratio at birth, and pregnant women aged 15–49 years who are anaemic. We evaluated the agreement between HMIS and NFHS data for the above indicators using Pearson’s correlation co-efficient, intraclass correlation coefficient, and Bland–Altman plot.

Results:

Data were available from both HMIS and NFHS-5 for 695 districts for the above indicators, except for pregnant women with anaemia, for whom data were available for 564 districts. Pearson corelation co-efficient showed a strong correlation between the two datasets for institutional births and delivery by caesarean section, while weak to moderate correlations were observed for the other indicators. Intraclass correlation coefficient showed discordance between the two datasets, and poor agreement was observed between the data for sex ratio at birth and mothers with at least 4 antenatal visits in Bland–Altman plot.

Conclusion:

Poor agreement was observed between HMIS and NFHS data for certain indicators, and steps can be taken to improve the quality of HMIS data for these indicators.

Background

Health information system is one among the six core components or “building blocks” of health system described under the WHO framework. The collection of data from health and other relevant sectors, as well as the analysis and processing of these data into information for effective decision making, while ensuring quality, relevance, and timeliness, is the cornerstone of health information system. Effective health system policy development and implementation, governance and regulation, health research, human resources development, health education and training, service delivery, and financing are dependent on reliable information across health systems. The WHO has stressed upon the need for availability of high-quality data from Health Monitoring Information System (HMIS) and other programme related data for monitoring the progress towards sustainable development goals.[1]

The Ministry of Health and Family Welfare (MoHFW) has established National Health Mission (NHM) and initiated various national programmes to improve the health services provided to the under-served population and address specific health problems.[2] HMIS is a web-based monitoring system established by MoHFW to monitor the implementation of activities under NHM and other national programmes.[2] Data obtained through HMIS have also been utilised for formulation of health policies, reviewing the activities under programme implementation plan (PIP)s, and decision making for evidence-based course correction of activities under various programmes.[2,3]

Facility level data obtained through HMIS may have quality limitations, and global report on health data and systems have stressed upon the need for improving data quality with quality assurance systems not documented in about 40% countries.[1] Regarding HMIS in India, various mechanisms are in place to ensure data quality, such as 1) inbuilt quality assurance mechanisms like compare options and inter-data validation checks and 2) external quality assurance mechanisms like ADV tool. These mechanisms try to address the completeness and internal consistency dimensions of data quality.[2,3]

Studies have been conducted in various low- and middle-income countries, such as Ethiopia[4,5,6] and Chad,[7] to assess the data quality of HMIS by conducting facility level surveys that assessed completeness, timeliness, and accuracy of data. Very few studies have been conducted to assess the external consistency of HMIS data. One such study conducted in Bihar[8] used annealing technique and provided a coverage estimator by combining data from a probability survey and HMIS data; in another study conducted in Ethiopia, HMIS data indicators were compared with a population survey indicator.[9]

Therefore, in this study, we aimed to assess the level of external consistency of HMIS data by comparing it with National Family Health Survey-5 (NFHS-5) data at district level across India from 2014 to 2019.

Methods

A cross-sectional study was conducted using district level data from HMIS and NFHS-5 covering all districts in India.

Data source

HMIS captures facility level and sub-centre level data on health service delivery related to reproductive, maternal, and child health, immunisation, family planning, vector borne disease, tuberculosis, morbidity and mortality, OPD and IPD services, and surgeries. Entries are made by the health care providers at primary health centre (PHC) and health sub centre (HSC) level using a separate login every month. Cumulative reports on the above services are available year wise for every state, up to the sub-district level, in the MoHFW website for HMIS.

NFHS is a large-scale, multi-round survey conducted in a representative sample of households throughout India. The survey provides national-, state-, and district-level information on fertility, infant and child mortality, the practice of family planning, maternal and child health, reproductive health, nutrition, anaemia, utilisation and quality of health, and family planning services. District-level estimates for the recent round of the survey (NFHS-5) is available for the above indicators in the factsheets published in the nfhs.in website.

Some indicators are common to both HMIS and NFHS, and data for these indicators were abstracted from both the sources. The indicators for which data abstraction was done are

1. Mothers registered within 12 weeks/trimester I

2. Mothers who had at least 4 antenatal visits

3. Mothers last birth protected against neonatal tetanus

4. Institutional births

5. Births delivered by Caesarean section

6. Sex ratio at birth

7. Pregnant women aged 15–49 years who are anaemic.

The above indicators, except for sex ratio at birth and pregnant women aged 15–49 years who are anaemic, are part of the composite indicators used for HMIS dashboard analysis, which is used to compare the performance at the district level. Ascertaining the consistency of these indicators will make the comparisons between districts more reliable and thereby helpful in making policy decisions regarding various reproductive, maternal, newborn, child, adolescent health and nutrition (RMNCAH+N) interventions.

Data analysis

Data cleaning was done, and data quality and completeness were checked using Microsoft excel. Districts were grouped based on the performance in the above indicators and described as frequency with percentage for each quartile. Scatter plot was generated to study the correlation between HMIS and NFHS data for the above indicators, and Pearson correlation co-efficient was calculated. Intraclass correlation (ICC) was calculated assuming a two-way mixed model, testing for absolute agreement between the two methods. A two-way mixed model was chosen, as the reliability was tested for the specific two methods for all districts. A Bland–Altman plot was created for each of the above indicators using the mean difference and average of the above two datasets to assess agreement between the two datasets and estimate the concordance between values from the HMIS and NFHS.

Ethical considerations

This study was conducted using district-level data from open source (websites of NFHS and HMIS) and did not directly involve data collection from participants. Ethical waiver was obtained from Institutional Ethics Committee of Madras Medical College.

Results

In NFHS-5, district-level data are available for the above indicators, except “Pregnant women aged 15–49 years who are anaemic,” for 704 districts covering 28 states and 6 union territories. Data for “Pregnant women aged 15–49 years who are anaemic” are available for 571 districts in NFHS-5. In HMIS, district-level data for the above indicators are available for 702 districts. Overall, district-level data were available for 695 districts in both NFHS-5 and HMIS for all the above indicators, except “Pregnant women aged 15–49 years who are anaemic,” for which data were available for 564 districts in both the above databases. These 695 districts, for all the 6 indicators, and 564 districts, for the indicator “Pregnant women aged 15–49 years who are anaemic,” were included in the final analysis.

Table 1 shows the comparison of district performance in the selected maternal and child health (MCH) indicators as per NFHS-5 and HMIS data. District performance reported by NFHS-5 and HMIS significantly differ for all the indicators except institutional births.

Table 1.

Comparison of district performance in the MCH indicators as per NFHS-5 and HMIS in 695 districts in India

Indicator	Data source	Performance of districts (%) (n=695)				Chi-square value	P
		<25%	26–50%	51–75%	>75%
Early AN registration	NFHS-5	-	44 (6.3%)	326 (46.9%)	325 (46.8%)	58.84	<0.001
	HMIS	6 (0.9%)	133 (19.1%)	291 (41.9%)	265 (38.1%)
4 antenatal visits	NFHS-5	27 (3.9%)	192 (27.6%)	288 (41.4%)	188 (27.1%)	173.1	<0.001
	HMIS	13 (1.8%)	47 (6.8%)	237 (34.1%)	398 (57.3%)
AN mother immunised with TT	NFHS-5	-	-	17 (2.5%)	678 (97.5%)	78.69	<0.001
	HMIS	4 (0.6%)	33 (4.8%)	73 (10.5%)	585 (84.2%)
Institutional births	NFHS-5	1 (0.1%)	13 (1.9%)	66 (9.5%)	615 (88.5%)	2.142	0.543
	HMIS	2 (0.3%)	9 (1.3%)	79 (11.4%)	605 (87.1%)
Caesarean sections out of total deliveries	NFHS-5	456 (65.6%)	185 (26.6%)	49 (7.1%)	5 (0.7%)	15.92	0.001
	HMIS	514 (74.0%)	149 (21.4%)	32 (4.6%)	-
Pregnant women aged 15–49 years who are anaemic (n=564)	NFHS-5	30 (5.3%)	227 (40.3%)	291 (51.6%)	16 (2.8%)	194.6	<0.001
	HMIS	79 (14.0%)	143 (25.4%)	175 (31.0%)	167 (29.6%)
Indicator	Data source	Females per 1000 males				Chi-square value	P
		<800	800–949	950-1200	>=1200
Sex ratio at birth	NFHS-5	61 (8.8%)	321 (46.2%)	294 (42.3%)	19 (2.7%)	185.9	<0.001
	HMIS	-	546 (78.6%)	149 (21.4%)	-

AN=Antenatal, HMIS=Health management information system, MCH=Maternal and child health, NFHS=National family health survey, TT=Tetanus toxoid

Correlation between HMIS and NFHS-5 for indicators at the district level

Figure 1 shows the scatter plot between HMIS and NFHS estimates for the selected MCH indicators across 695 districts. Pearson correlation coefficient shows that HMIS and NFHS-5 estimates are weakly correlated for sex ratio at birth (r = 0.148), pregnant women aged 15–49 years who are anaemic (r = 0.135), early antenatal registration (r = 0.356), and antenatal mothers immunised with TT (r = 0.225), while a moderate correlation was observed between the two estimates for antenatal mothers having at least 4 antenatal visits (r = 0.532). HMIS and NFHS-5 estimates were strongly correlated for institutional births (r = 0.758) and Caesarean section out of total deliveries (r = 0.892) [Table 2].

Figure 1.

Scatter plot showing correlation between HMIS and NFHS-5 estimates for the selected MCH indicators across 695 districts in India. (a) Sex ratio at birth (r = 0.148), (b) Early AN registration (r = 0.356), (c) Mothers with minimum 4 antenatal visits (r = 0.532), (d) Antenatal mothers immunised with TT (r = 0.225), (e) Institutional births (r = 0.758), (f) Caesarean section out of total births (r = 0.892), (g) Pregnant women aged 15–49 years who are anaemic (r = 0.135)

Table 2.

Pearson correlation coefficient and intraclass correlation coefficient for relationship between HMIS and NFHS-5 estimates for MCH indicators across 695 districts in India

Indicator	Pearson correlation coefficient	Intraclass correlation coefficient (95% CI)
Sex ratio at birth	0.148	0.066 (-0.007–0.138)
Early AN registration	0.356	0.328 (0.248–0.401)
4 antenatal visits	0.532	0.411 (0.084–0.614)
AN mother immunised with TT	0.225	0.131 (0.051–0.208)
Institutional births	0.758	0.753 (0.719–0.784)
Caesarean sections out of total deliveries	0.892	0.828 (0.456–0.922)
Pregnant women aged 15–49 years who are anaemic (n=564)	0.135	0.084 (0.005–0.162)

AN=Antenatal, CI=Confidence interval, HMIS=Health management information system, MCH=Maternal and child health, NFHS=National family health survey, TT=Tetanus toxoid

Intraclass correlation coefficient values for all the indicators except institutional births and Caesarean sections out of total deliveries were below 0.5 and close to zero, which indicate poor reliability and large degree of discordance between HMIS and NFHS-5 estimates [Table 2].

Bland–Altman analysis for agreement between HMIS and NFHS-5 estimates for indicators at the district level

Bland–Altman plot analysis showed that on average, estimates reported through HMIS were higher than the estimates from NFHS-5 for pregnant mothers having minimum 4 antenatal visits, institutional births, and pregnant women aged 15–49 years who are anaemic, whereas the HMIS values were lower than NFHS-5 estimates for sex ratio at birth, early antenatal registration, antenatal mothers immunised with TT, and Caesarean section out of total births [Table 3]. Concordance between the two estimates was higher for institutional births, while poor agreement was noted between the two methods for sex ratio at birth and pregnant mothers having minimum 4 antenatal visits. Line of equality was not included in the 95% confidence interval (95% CI) of mean difference for any of the indicators, which indicates a significant systematic difference between the two methods [Figure 2].

Table 3.

Bland–Altman summary statistics for the analysis of agreement between HMIS and NFHS-5 survey estimates across 695 districts in India

Indicators	Mean difference (%) 95% CI	95% limits of agreement (2.5th & 97.5th percentile)	Number of potential outliers
Sex ratio at birth (Females per 1000 males)	16 (11.2 to 20.8)	-111, 143	181
Early AN registration	5.0 (3.6 to 6.3)	-30.2, 40.1	31
4 antenatal visits	-15.1 (-16.6 to -13.5)	-54.9, 24.8	25
AN mother immunised with TT	6.5 (6.0 to 7.0)	-24.5, 37.5	38
Institutional births	-0.9 (-1.5 to -0.2)	-18.1, 16.3	42
Caesarean sections out of total deliveries	6.3 (5.8 to 6.9)	-8.3, 20.9	39
Pregnant women aged 15–49 years who are anaemic (n=564)	-10.7 (-14.0 to -7.5)	-87.8, 66.4	14

AN=Antenatal, CI=Confidence interval, HMIS=Health management information system, NFHS=National family health survey, TT=Tetanus toxoid

Figure 2.

Bland–Altman plot for the analysis of agreement between HMIS and NFHS-5 survey estimates across 695 districts in India. (a) Sex ratio at birth, (b) Early AN registration, (c) Mothers with minimum 4 antenatal visits, (d) Antenatal mothers immunised with TT, (e) Institutional births, (f) Caesarean section out of total births, (g) Pregnant women aged 15–49 years who are anaemic

Discussion

This cross-sectional study, using district-level data on selected MCH indicators from HMIS and NFHS-5, showed that the data from the two methods were weakly correlated for all indicators, except for institutional births and Caesarean section out of total births, and have a large degree of discordance. Bland–Altman plot also showed the presence of significant systematic difference between the two methods.

A study was conducted in Jimma Zone of Ethiopia to assess the quality of HMIS data using all the data quality dimensions of WHO data quality report card. In this study, the external consistency of coverage rates of certain MCH indicators reported under HMIS was assessed by comparing with data from a cross-sectional survey done in the same region to assess the barriers to safe motherhood. Similar to our study, the HMIS and survey data were discordant to a large extent in the above study as per ICC. The study also found a poor agreement overall between the two methods using Bland–Altman analysis, while there was a good agreement for certain indicators such as still birth rate in all districts.[9] Similar to our study, a study comparing the routine health information system data with population level health survey in Mozambique showed strong correlation between the two methods for institutional births.[10]

Similar to our study, in an analysis of HMIS data quality in Ethiopia for a period of 6 years, external consistency was checked by comparing with data obtained in Ethiopia district health survey (DHS) for 19 indicators related to MCH and communicable diseases.[11] Though other indicators compared in the above study were different from the indicators analysed in our study, one indicator—antenatal mothers receiving 4 antenatal care visits—was overreported in HMIS compared to DHS data in the Ethiopian study, aligning with the findings of our study.

A national level analysis of data from countries in Eastern and Southern Africa showed that nearly half of the institutions reported extreme outliers in the coverage rates for antenatal (AN) care, immunisation, and OPD services, and only about one-third of units had good internal consistency for the indicators.[12] In a similar study in Rwanda, the completeness of HMIS data increased over the years to above 95%, and there were no moderate or extreme outliers at district level, and good internal consistency was observed over the years.[13]

There are very few studies in India evaluating the quality of HMIS data. One such study done using annealing technique to compare block-level HMIS data with a sample survey using lot quality assurance sampling (LQAS) in Bihar showed discordance between the HMIS coverage rates and estimates from the sample survey.[8] The above study evaluated 10 indicators related to MCH; four of these indicators are similar to those in our study: 1) first trimester registration, 2) having 3 or more AN visits, 3) received TT immunisation, and 4) institutional births. In the above study, discordance was found between HMIS data and sample survey data for all the MCH indicators, although our study showed high correlation between the survey estimates and HMIS data for institutional births.[8]

There are studies from other countries evaluating the data quality of HMIS in terms of completeness, timeliness, and accuracy through record review. One such study at a township level in a region in Myanmar showed that although the timeliness of the reporting was good, data accuracy was low.[14] Another such study in 14 health facilities in Massaguet district, Chad, also showed similar results, with a high degree of data completeness but low accuracy. In a study in Southern Ethiopia, completeness and timeliness of HMIS reporting were above 80%; however, data accuracy was within the acceptable level in only 44% of facilities, with all facilities over reporting coverage data in HMIS compared to the records in the facilities.[5] In another study in Hadiya zone of Ethiopia, completeness and timeliness of data were above 80%, and data accuracy was also above 75%.[4]

Our study is the first one to assess the external consistency of HMIS data in India and included data for nearly all the districts across the country, which improves the generalizability of the findings. Survey data used for comparison are from NFHS, which is a nationwide survey and includes robust sampling strategy covering more than 6 lakh households and 7 lakh women with adequate representations from each district. Therefore, data variability across the districts is expected to be minimal, making the comparison more generalizable.

Our study did not assess other attributes of data quality outlined in WHO data quality report card which is a limitation. HMIS incorporates mechanisms to verify the internal consistency of the indicators within the platform. The time of report submission is not available, and data are available up to the level of subdistrict only in the HMIS platform in open source; hence, the timeliness and completeness of health facility reporting could not be assessed. The definitions of certain indicators such as anaemia may vary between the two survey methods, and the training of persons reporting in HMIS and those involved in the survey vary, which may have led to some discordance between the data.

This study concludes that there is discordance between data reported through HMIS and NFHS estimates for certain MCH indicators, except for institutional deliveries and Caesarean section out of total deliveries. Further studies may be done to assess the accuracy of HMIS data by comparing with records at the health facilities. Studies to explore the challenges faced by health workers in reporting data through HMIS may also be helpful. Periodical training of health workers in HMIS reporting can also be performed to minimize the data errors.

Conflicts of interest

There are no conflicts of interest.

Funding Statement

Nil.