A Review of Data Quality of an Electronic Tuberculosis Surveillance System for Case-based Reporting in Kenya

Abstract

Kenya recently transitioned from a paper to an electronic system for recording and reporting of tuberculosis (TB) data. During September–October 2013, the data quality of the new system was evaluated through an audit of data in paper source documents and in the national electronic system, and an analysis of all 99,281 cases reported in 2012. While the new electronic system overall is robust, this assessment demonstrated limitations in the concordance and completeness of data reaching the national level. Additional oversight and training in data entry are needed to strengthen TB surveillance data quality in Kenya.

Introduction

Kenya is one of the 22 high TB burden countries worldwide as classified by the World Health Organization (WHO), with an estimated annual incidence of 272 cases per 100,000 persons¹. In 2012, almost 100,000 new and relapse cases were reported nationally.

A major objective of the Stop TB Strategy is the enhancement of high-quality DOTS programs². To achieve this objective, WHO recommends establishing and maintaining well-developed monitoring and evaluation systems within national TB programs. Compared to paper systems, electronic recording and reporting systems can strengthen data quality through automated validation checks, accelerate the availability of data for managers and decision-makers, and facilitate the analysis of local and national trends to monitor progress in TB control³.

In 2012, the Division of Leprosy, Tuberculosis and Lung Disease, Ministry of Health, Kenya underwent the transition from a paper-based recording and reporting system to an electronic system called Treatment Information from Basic Unit, or TIBU⁴. TIBU is a national case-based surveillance system that stores details on individual patient episodes of TB reported to the national TB program. It includes patient locating information, demographic characteristics, clinical details, laboratory results, and treatment outcome data. TIBU has had nationwide coverage since 2012⁵.

The flow of information in the surveillance system is based on a hierarchical reporting structure⁶. At the facility level (e.g. dispensaries, clinics, and hospitals), health workers record key data on TB patient cards, which are then transcribed into TB facility registers. During monthly visits to patient-care sites in jurisdictions called TB reporting zones, TB coordinators electronically transcribe data from TB registers using tablet computers provided by the national TB program. The tablet computers connect to the internet through Kenya’s mobile data network and automatically transmit encrypted case-based data in real-time to a password-protected national server. TIBU data can be reviewed by TB coordinators to improve local case management and by staff at the national TB program for population-level review and analysis.

To help inform the national TB program’s ongoing efforts to improve the new electronic system, we conducted a review of data captured by TIBU by analyzing concordance between source documents at TB facilities and TIBU, and by measuring the completeness of data transcribed from TB facility registers to TIBU for all cases registered in 2012.

Methods

For analysis of concordance, we visited two facilities from each of six randomly selected TB reporting zones located in four provinces near Nairobi. In total, 5 hospitals, 4 health centers, and 3 dispensaries/clinics were visited, with an average of 129 (range 6 – 311) cases per facility. All cases recorded in TB facility registers in the first quarter of 2012 were reviewed. Data for these cases in TB facility registers and corresponding TB patient cards were assessed for concordance with data in TIBU using Fleiss' Kappa coefficient. Calculation of concordance accounted for data recorded in one source and not recorded in another.

To assess completeness, 99,281 cases registered nationwide for treatment in 2012 and recorded in TIBU were reviewed. The case reports were de-identified and provided in electronic format by the national tuberculosis program. Completeness for key data items were calculated using proportions.

CDC and the Kenya Ministry of Health both reviewed and approved this activity and determined that it was part of routine program evaluation and not human subjects research.

Results

Of the 222 cases reviewed from TB facility registers, 44 (20%) were located at dispensaries/clinics, 80 (36%) at health centers, and 98 (44%) at hospitals. Only 152 (69%) cases had a corresponding TB patient card at the site of care delivery, and 219 (99%) were present in TIBU. Of 70 cases with missing TB patients cards, 50 (71%) were located at hospitals, 16 (23%) at health centers, and 4 (6%) at dispensaries/clinics.

Table 1 demonstrates that the patient cards and the registers were mostly concordant, with kappa scores above 0.60 for most variables. The concordance between the registers and TIBU was better than that between TB patient cards and TIBU, although the district registration number and the registration date had kappa scores below 0.40.

Table 1.

Concordance of key variables from TB Patient Card, TB Facility Register, and Treatment Information from Basic Unit (TIBU), κ coefficient^*

		Concordance with TB Patient Card (n = 152)		Concordance with TB Facility Register (n = 222)
Category	Variable	TB Facility Register	TIBU	TIBU
Demographic	District registration number	0.81	0.26	0.27
	Age	0.95	0.91	0.96
	Sex	0.93	0.91	0.97
Clinical	Registration date	-	-	0.21
	Type of tuberculosis	0.83	0.86	0.95
	Type of patient	0.58	0.60	0.76
	HIV status	0.64	0.57	0.79
	ART	0.57	0.54	0.85
	CPT	0.75	0.75	0.94
Bacteriology	Sputum at 0 months	0.77	0.66	0.74
	Sputum at 2/3 month	0.84	0.58	0.64
	Sputum at 6 months	0.87	0.46	0.51
	Sputum at 8 months	0.74	0.47	0.612
Drug susceptibility	Culture	0.71	0.63	0.77
Treatment	Treatment start date	0.81	0.59	0.65
	Treatment outcome	0.13	0.16	0.68
	Treatment end date	0.63	0.39	0.48
Treatment data	Referred by	0.27	0.30	0.78
	Referred to	0.44	0.26	0.50
	Nutrition support	0.01	−0.13	0.40
	DOT provider	0.23	0.22	0.54

^*κ values range from 1 to −1, with a value of 1 indicating 100% agreement between sources, a value of 0 indicating no agreement beyond what one would expect by chance, and a value of −1 indicating 100% disagreement. A value of 0.8–1.0 can be considered almost perfect agreement, 0.6– 0.8 reflects substantial agreement, 0.4– 0.6 moderate agreement, 0.2– 0.4 fair agreement, 0– 0.2 slight agreement, and <0 reflects poor agreement.

A review of the completeness of the national 2012 data showed mixed results. Over 99% of cases had complete fields for treatment start date, age, sex, type of TB patient, site of disease, and initial sputum smear result. However, a high percentage of cases had blank values for HIV test date (47%), treatment regimen (12%), and treatment outcome date (30%).

Discussion

Kenya is the first country in East Africa to implement a national case-based electronic surveillance system for tuberculosis, permitting quicker reporting and analysis compared to a paper-based system. Making the transition from paper to electronic recording and reporting is complex, requiring an advanced information technology infrastructure and significant investment in human resources³. However, the improved timeliness of TIBU is counterbalanced by limitations in data quality. A successful rollout of a nationwide electronic surveillance system should be accompanied by routine review of data and regular feedback to users to improve the quality of data captured by the system.

Data in TB facility registers had higher concordance with TIBU than TB treatment cards, which is consistent with the practice of using TB facility registers as the primary record for documenting the treatment course of TB patients. Several variables had marginally higher concordance for TIBU and TB treatment cards compared to TIBU and TB facility registers. This result may be due to errors in transcription from treatment cards to the facility register that are corrected on entry into TIBU. The proportion of missing TB patient cards was highest among health centers and hospitals, suggesting that higher-level health facilities could benefit from training on record-keeping. The low concordance of district registration numbers and registration dates between TB facility registers and TIBU suggests a systematic error in the transcription of these variables. Regular data audits and data validation rules can detect and minimize these types of systematic errors.

This evaluation demonstrated two major implications of using TIBU for TB surveillance in Kenya. First, the suboptimal completeness and concordance of data may hinder informed decision-making at the national level and may present challenges in using TIBU data to facilitate case management by TB district coordinators. Second, some cases from TB facility registers were not found in TIBU, suggesting that some reports were not transcribed to TIBU or were transcribed inaccurately. A nationwide inventory study is needed to better understand the extent of underreporting⁷.

Because the modest number of facilities that were evaluated might not be representative, results may not reflect TIBU’s concordance with source documents nationwide. Additionally, the study could not assess concordance stratified by facility type. Although the evaluation found aspects of TIBU that should be strengthened, the completeness and concordance of TIBU are comparable to findings from an evaluation of a well-established electronic tuberculosis reporting system (ETR.Net) in South Africa⁸.

Electronic recording and reporting systems can greatly improve the efficiency of data collection, aggregation, and analysis for national TB programs. With TIBU, Kenya has demonstrated that a robust electronic surveillance system can be used successfully for disease surveillance in a resource-limited setting. Countries seeking to implement electronic surveillance systems to monitor and evaluate disease trends should establish mechanisms to regularly evaluate data quality and provide feedback to users in order to optimize impact on TB control and prevention.

Keypoints.

• Kenya Ministry of Health recently deployed a case-based national electronic surveillance system for tuberculosis.

• Overall the new electronic surveillance system is robust but improvement in concordance and completeness of data is needed.

• Additional oversight and training in data entry and record management may strengthen TB surveillance data quality in Kenya.