Improving public health information: a data quality intervention in KwaZulu-Natal, South Africa

Abstract

Objective

To evaluate the effect of an intervention to improve the quality of data used to monitor the prevention of mother-to-child transmission (PMTCT) of the human immunodeficiency virus in South Africa.

Methods

The study involved 58 antenatal clinics and 20 delivery wards (37 urban, 21 rural and 20 semi-urban) in KwaZulu-Natal province that provided PMTCT services and reported data to the District Health Information System. The data improvement intervention, which was implemented between May 2008 and March 2009, involved training on data collection and feedback for health information personnel and programme managers, monthly data reviews and data audits at health-care facilities. Data on six data elements used to monitor PMTCT services and recorded in the information system were compared with source data from health facility registers before, during and after the intervention. Data completeness (i.e. their presence in the system) and accuracy (i.e. being within 10% of their true value) were evaluated.

Findings

The level of data completeness increased from 26% before to 64% after the intervention. Similarly, the proportion of data in the information system considered accurate increased from 37% to 65% (P < 0.0001). Moreover, the correlation between data in the information system and those from facility registers rose from 0.54 to 0.92.

Conclusion

A simple, practical data improvement intervention significantly increased the completeness and accuracy of the data used to monitor PMTCT services in South Africa.

Introduction

Reliable and accurate public health information is essential for monitoring health and for evaluating and improving the delivery of health-care services and programmes.^1–4 As countries report their progress towards achieving the United Nations Millennium Development Goals, the need for high-quality data has never been greater.^5,6 Furthermore, funding and support for public health activities, such as immunization programmes, remain contingent on demonstrating coverage using routine statistics.⁷ However, assuring the quality of health information systems remains a challenge.

Studies of public health information systems in resource-poor countries frequently document problems with data quality, such as incomplete records and untimely reporting.^8,9 Yet these systems are often the only data sources available for the continuous, routine monitoring of health programmes.^10,11 Efforts have been made to improve the quality and management of public health information systems in developing countries. Two examples are the Health Metrics Network, an international network that seeks to improve the quality of health information from various sources,¹² and the Performance of Routine Information System Management (PRISM) framework, which was developed as a method for assessing the strengths and weaknesses of routine health information systems.^13,14 Other initiatives, such as the Data Quality Audit, have been used by the GAVI Alliance to improve the monitoring of immunization coverage.⁷ However, the complex nature of health information systems and the demands placed upon them have complicated efforts to improve the quality of routine data.¹⁵

In South Africa, the effect of human immunodeficiency virus (HIV) infection on maternal and child health has raised considerable concern. The reported prevalence of HIV infection among women attending antenatal clinics in the province of KwaZulu-Natal, for example, is 38.7%, the highest in the country.¹⁶ This led South Africa to set a target of reducing the rate of mother-to-child HIV transmission to below 5% by 2011.¹⁷ Although HIV testing is essential for delivering interventions for the prevention of mother-to-child transmission (PMTCT) and thereby for reducing maternal and infant mortality,¹⁸ routinely available health information indicates that the HIV testing rate is highly variable.^19,20 The questionable accuracy and reliability of these data compromise efforts to improve the health-care systems that provide PMTCT interventions.²⁰

Routine health data from the primary health-care system in South Africa, which is organized in districts, are collected and stored in the District Health Information System (DHIS).²¹ Recent studies of this system, however, have reported that the quality of the data, including those used to track PMTCT care, is suboptimal and is hindering efforts to strengthen service delivery.^20,22 As a result, in 2008 the KwaZulu-Natal Department of Health, the University of KwaZulu-Natal and the Institute for Health care Improvement launched a large-scale effort, entitled the 20000+ Partnership, to improve the quality of PMTCT services in three health districts in the province. This programme included an intervention to increase the completeness and accuracy of the public health data routinely recorded in the DHIS. The aim of this paper is to report on the effect of that intervention.

Methods

Routine data collection for the DHIS in South Africa starts with information being collected in registers at each point where clinical care is provided. Every month the staff at the different primary health-care facilities collate the data and send monthly summaries on paper to a clinic supervisor who, on average, oversees six facilities. The monthly summaries are then converted into electronic format by an information officer based either at the facility or centrally within the district health office. Larger facilities such as community health centres and hospitals have dedicated facility information officers who themselves create electronic files containing the facility data. District information officers oversee the input of all data into the DHIS. In addition, district PMTCT coordinators work regularly with district information officers to ensure that all health-care workers receive training on data reporting requirements, data collection tools and any changes or updates to the definitions of the data elements used as indicators of the performance of PMTCT services.

Study design

The study involved data collected before and after the data quality improvement intervention at 58 antenatal clinics and 20 delivery wards in both urban and rural communities in KwaZulu-Natal. These 78 locations were randomly selected from 222 fixed facilities that provide PMTCT services in three of the 11 districts in the province: eThekwini, Umgungundlovu and Ugu districts. Across KwaZulu-Natal province, over 90% of pregnant women attend antenatal care facilities at least once during pregnancy and over 90% of deliveries take place in delivery wards at health-care facilities.^23,24 The prevalence of HIV infection among expectant mothers in the three districts ranged from 40.3% to 45.7% in 2008.¹⁷

The data quality improvement intervention took place between May and November 2008 and included three main components: (i) training days on data collection, (ii) monthly reviews of data and (iii) data audits at individual facilities.

Three data training days for health-care facility and district information officers and PMTCT programme managers took place in May, August and November 2008, respectively. Each one-day training session dealt with: standardizing data collection procedures; improving data quality; selecting data elements that were useful for monitoring the quality of data on PMTCT services; and providing clear definitions of those data elements. After training, each district information officer liaised on data quality with facilities in his or her district.

Monthly reviews of routine data on PMTCT services were carried out in each district by a PMTCT improvement task team, which comprised the district information officer, the PMTCT programme coordinator and the primary health-care coordinator. During these monthly meetings, the study team provided the task teams with information about the completeness and accuracy of the PMTCT data elements they were using to assess the PMTCT programme. Thereafter, each district task team provided feedback to the facilities on the quality of the data.

Data audits were carried out by study staff at the 78 randomly selected facilities three times: before, during and after the intervention. The original source data in facility registers covering the 3 months preceding the audits were collected and summarized in the same manner in which they were routinely dealt with at the end of each month. The accuracy of the process by which these data were collated at health-care facilities, transmitted to the district offices and entered into the DHIS was assessed by examining the degree of agreement between the data elements for a particular clinic that appeared in the DHIS and the data elements recorded by the study team for the same time period. It is important to note that whether or not the data contained in the registers accurately represented the clinical activities taking place at each facility was not assessed, as resources did not allow for direct clinical observation. After each audit, the study team provided individual facilities with feedback on the quality of its data on PMTCT services.

Ethical approval for the study was obtained from the University of KwaZulu-Natal Biomedical Research Ethics Committee. In addition, consent was obtained from the KwaZulu-Natal Department of Health and the district managers of all three participating districts. The funding agency, the Centers for Disease Control and Prevention in Atlanta, United States of America (USA), also gave approval but did not influence the study design or data reporting.

Data collection and assessment

The study team visited the 78 randomly selected facilities before the intervention between September and November 2007, during the intervention between June and September 2008 and after the intervention in March 2009.

At each visit, the team collected information on the six data elements used for monitoring PMTCT services that were routinely recorded in registers on the number of: (i) pregnant women who were tested for HIV antenatally; (ii) pregnant women who tested positive for HIV; (iii) pregnant women in whom CD4+ lymphocyte quantitation was performed; (iv) pregnant women who received nevirapine in the delivery ward; (v) babies who were born to an HIV-positive woman and who underwent a polymerase chain reaction test for HIV; and (vi) babies who received nevirapine. These six data elements were selected from a total of 24 possible elements because of their clinical importance. The characteristics of each facility were recorded in a simple form that was completed by the facility manager during the site visit.

It was possible to obtain information on the completeness of the data reported between January 2007 and March 2009 to the DHIS for all 222 facilities providing PMTCT services in the study districts. Completeness was quantified by determining whether a value for a given data element from a specific facility was or was not present in the information system. Data completeness was analysed by data element and facility, as well as over time.

Data accuracy was determined by comparing the data recorded in the DHIS with the data obtained from individual facility registers during data audits. An individual data element value was defined as accurate if the value recorded in the information system lay within 10% of the corresponding value “reconstructed” by the study team from the original source data during the data audit. The 10% threshold was based on that used in previous studies.^11,20 No assessment of the accuracy of a data element value was made if that value was missing from either the facility register or the DHIS or if the difference between the value reconstructed by the study team and that in the information system was greater than 1000%. Such large differences were regarded as “errors” rather than inaccuracies.

Statistical analysis

The data analysis was carried out using Excel 2007 (Microsoft, Redmond, USA). The change in data accuracy from before to after the intervention was evaluated using the χ² test, with an α of 0.05. The degree of agreement between data reported in the DHIS and data obtained from facility registers during the data audits was evaluated using Pearson’s correlation coefficient (r), both before and after the intervention.

Results

Of the 78 facilities randomly selected to participate in the study of the data quality intervention, 37 (47%) were urban sites, 21 (27%) were rural and 20 (26%) were semi-urban. The median number of clients seen daily at each site was 28 and the median number of health-care workers at each site was 6. All facilities had been offering PMTCT services for more than 2 years.

Data completeness

Overall data completeness improved from 26% before the intervention at all 222 facilities to 64% after: of the 1115 data element values that could have been completed in January 2007, 285 were completed, while of the 1010 that could have been completed in March 2009, 643 were completed. In the antenatal clinics, data completeness improved from 25% (i.e. 271 of 1065 relevant data element values were completed) to 63% (i.e. 600 of 960 relevant data element values were completed). In the delivery wards, data completeness improved from 28% (i.e. 14 of 50 relevant data element values were completed) to 86% (i.e. 43 of 50 relevant data element values were completed). Fig. 1 shows the change in completeness for the six data elements individually.

Fig. 1.

Completeness of data^a used to monitor the prevention of mother-to-child HIV transmission, before and after a data quality intervention, by data element, South Africa, 2007–2008

HIV, human immunodeficiency virus.

^a Data completeness was quantified by determining whether a value for a given data element from one of 222 health-care facilities was or was not present in the District Health Information System.

Data accuracy

Overall, 55% of data element values were excluded from the accuracy analysis before the intervention because of errors or missing data, compared with 37% after. The overall accuracy of the remaining data on all six data elements improved from 37% at the first data audit to 65% at the third data audit (P < 0.0001). In antenatal clinics, data accuracy improved from 41% to 65% (P < 0.0001); in delivery wards, it improved from 32% to 64% (P < 0.0001).

During the first data audit before the intervention, there was poor correlation between data that appeared in the DHIS and data obtained from facility registers (r = 0.54, Fig. 2). The correlation was much better for data obtained after the intervention during the third data audit (r = 0.92, Fig. 2).

Fig. 2.

Correlation between data element^a values observed on data audit and those held in the District Health Information System (DHIS), before the data quality intervention, South Africa, 2007 and 2008

HIV, human immunodeficiency virus.

^a Six data elements were used to monitor a programme for the prevention of mother-to-child HIV transmission at each of 78 health-care facilities.

Discussion

A data quality improvement intervention that involved specific training for health-care workers on the importance of public health information, monthly data reviews and feedback, and regular data audits was effective in significantly increasing the completeness and accuracy of the data used to monitor PMTCT services in South Africa.

Training principally involved the health-care facility and district information officers who were responsible for data collection and was intended to help them understand that high-quality data are essential for improving PMTCT programme performance. It was possible to use monthly district programme review meetings, which were often already under way, as an opportunity to reflect on data quality. In the study, data audits were carried out by a small team of four data clerks. Although we did not undertake a formal costing analysis, these activities are likely to be a relatively inexpensive way of improving data quality in resource-poor settings before more systemic or expensive solutions, such as computerized data management, can be implemented.

Although previous reports have made recommendations for improving public health information, few studies have gone further than describing the inadequacy of the information available.^11,20,22,25 One systematic review identified 12 studies of activities for improving data quality: all were conducted in well resourced health-care services and only one appeared to be an interventional study.^26,27 However, three other studies reported the outcomes of data quality improvement interventions in resource-limited settings, in Kenya, Malawi and Peru. In Peru, phone reminders to epidemiological surveillance teams and clinic visits were used to improve the timeliness and accuracy of data reported in an electronic surveillance system of infectious disease outbreaks used by the navy. Phone reminders but not clinic visits improved timeliness, whereas in some settings data accuracy was improved by visits but not phone reminders.²⁸ In Malawi, the completeness and accuracy of public health information on HIV testing were reported, though not quantified, using a software package provided by a donor-funded HIV programme.²⁹ In Kenya, investigators used on-site assessment and feedback on data completeness and validity to improve the quality of routine health information: the number of HIV tests recorded and the number of HIV positive patients identified both doubled.³⁰ Finally, a recent study found that sequential data quality audits in six countries improved the quality and accuracy of data on immunization,⁷ which confirms our finding that regular audits can help improve data quality.

Despite the improvement in data quality observed during this study, the absence of a comparison study arm limits our ability to conclude that there was a strong cause-and-effect relationship with the intervention. Part of the improvement may have resulted from the stringent scrutiny that the South African PMTCT programme received from both programme donors and the department of health. However, we know of no specific efforts to improve data quality in the study facilities or districts at the time of the intervention. The well known inaccuracy of the DHIS has led many nongovernmental organizations, academic bodies and other development partners in South Africa to adopt parallel data collection systems to monitor PMTCT programme performance. These systems, while effective in serving the specific needs of these organizations, were not designed to improve the quality of the information system. An additional limitation of our study is that personnel at participating districts often retrospectively inserted missing data and corrected major errors in the DHIS for up to 1 year. These corrections could have made the data that were available at baseline appear to be better than they actually were and could have resulted in an underestimation of the effect of the intervention.

One unexpected complication that arose during the study could have reduced the effect of the data improvement intervention. The PMTCT programme in South Africa is relatively dynamic and the names and definitions of the data elements used for monitoring are frequently changed. Several challenging changes occurred during the study. For example, the data element used in the DHIS to record whether a baby had undergone a polymerase chain reaction test for HIV at 6 weeks was initially titled “HIV 1st test of baby born to HIV-positive woman” but was later changed to “HIV PCR test of baby born to HIV-positive woman at 6 weeks or later”. Such changes were made without the district offices providing definitions to the clinics. This could have caused considerable confusion at individual facilities and compromised the quality of reporting on that particular data element. To lower the chance of a similar occurrence we selected data elements whose descriptions had remained largely unchanged in recent years.

Despite these limitations, the improvement in PMTCT data quality observed in this study is encouraging, for it suggests that similar approaches could improve the quality of the data needed for decision-making and resource allocation in other public health programmes. The rationalization of data collection tools, clear definitions of data elements, continuous feedback on data quality and intermittent but regular data audits are effective ways of improving data quality. However, while this study shows that public health information can be improved, the final result falls short of what we should accept from our health information systems. The quality of public health data could be improved further. In addition, more work is needed to ensure that the data transmitted from bedside to the public health record are complete and accurate and that the original data collected in consultation rooms accurately reflect the interaction between patient and health-care provider.

In conclusion, this study reports a simple, practical approach to improving the quality of public health information, both locally and in a district health information system. Accurate information is vital for health-care planning and for evaluating progress towards specific health targets. Further research is needed to assess the effectiveness of similar interventions on a large scale, to estimate their cost implications and to clarify the connection between improved data quality and better health service delivery.