Sensitivity, completeness and agreement of the tuberculosis electronic system in Ho Chi Minh City, Viet Nam

Abstract

Setting: Since 2011, tuberculosis (TB) clinics in Ho Chi Minh City (HCMC), Viet Nam, have been entering data from a paper-based TB treatment register into an electronic database known as VITIMES (Viet Nam TB Information Management Electronic System), which is currently used in parallel with the paper system.

Objective: To evaluate the sensitivity, completeness and agreement of data in VITIMES with that of paper-based registers among TB patients co-infected with the human immunodeficiency virus (HIV) being treated for TB in HCMC.

Design: This was a retrospective data review of all TB-HIV patients receiving anti-tuberculosis treatment in each of the 24 district TB clinics in HCMC in 2013. Data were abstracted from the paper-based TB treatment registers at district level and extracted electronically at the provincial level. Records were matched based on name, age and address. The sensitivity, completeness and agreement of the electronic data were compared with data from the paper system.

Results: The findings showed that the electronic system had high sensitivity (99.2%), high completeness (87–99%) and high agreement (κ 0.78–0.97) for all variables.

Conclusion: The results of this study suggest that data are being correctly entered into VITIMES and that patient data can be directly entered into VITIMES instead of having a parallel, paper-based system.

Viet Nam ranks fifteenth among the 20 high tuberculosis (TB) burden countries in terms of absolute numbers of incident cases, as defined by the World Health Organization (WHO).¹ The estimated TB incidence (including human immunodeficiency virus [HIV] positive TB patients) in 2015 was 137 per 100 000 population, the incidence of TB and HIV coinfection was estimated at 5.9/100 000 and HIV prevalence among TB patients was 4.3%.¹ The province of Ho Chi Minh City (HCMC) has the greatest number of TB and HIV cases in Viet Nam. HCMC reported 16 219 TB cases in 2015, accounting for 16% of the national TB burden, and 49 561 HIV cases, accounting for 22% of the total number of people living with HIV/AIDS (acquired immune-deficiency syndrome) in Viet Nam.

Each TB clinic maintains a paper-based TB treatment register that is used to track patients undergoing anti-tuberculosis treatment. The TB treatment register is part of the national paper TB recording and reporting system, and consists of several components, including the TB treatment card, quarterly reporting forms and TB referral/transfer forms. Each clinic reports selected indicators from the register to the province on a quarterly basis in aggregate form. At the provincial level, data from all district clinics are further aggregated and forwarded to the National TB Programme (NTP) in Hanoi. In 2009, the NTP launched the Viet Nam TB Information Management Electronic System (VITIMES), an electronic web-based, case-based system, with the goal of improving the timeliness of TB surveillance and case management procedures with quarterly electronic reports on TB case registration, TB treatment outcome, sputum smear microscopy conversion and monitoring of TB drug supplies.

VITIMES was implemented in a phased approach. In Phase I, VITIMES was implemented in 2010 in all 63 provinces at the provincial level, where aggregate data were entered into VITIMES from the quarterly district paper TB reports. In Phase II, individual patient information was entered at the clinic (district) level by trained TB staff. By 2015, Phase II had been implemented in over 90% of the 701 districts nationwide. Provincial programmes can access VITIMES data from all districts within the province. Since 2011, all 24 district TB clinics in HCMC have used VITIMES in parallel with the paper TB treatment register. Currently, both the paper-based and electronic reporting systems are being used, and the paper TB treatment register is used as the main source document for inputting data into VITIMES.

In Viet Nam, TB and HIV services are provided through independent, vertical programmes. Although there are mechanisms for data sharing between the two programmes at all levels (Figure), actual data sharing has varied across provinces and districts. Consequently, integration of data on HIV status and treatment for patients in the TB system and on TB diagnosis and treatment for patients in the HIV system is challenging. Although in 2012 the Viet Nam Ministry of Health issued Decision No 2496, recommending collaborative activities, including data sharing, between the two programmes, no standard referral sheet or form was developed to allow for the exchange of essential clinical information between HIV and TB care facilities. Instead, each programme designed its own referral form. Given this separation of care, data may be more easily updated in the paper TB treatment register without updating the data in VI-TIMES accordingly. We therefore chose to assess the data on HIV-infected TB patients to gauge the quality of the data among a subset of records that were more likely to have incomplete data. Specifically, we sought to evaluate the completeness of data in the paper register, particularly with regard to the HIV variables, and then to evaluate the sensitivity of VITIMES, the completeness of data and the agreement of the data in VITIMES with those in the paper TB treatment register in HCMC.

FIGURE.

Surveillance data flow of TB and HIV registry in Viet Nam. Note: Data sharing between TB and HIV programmes (although mechanisms for data sharing exist, actual data sharing has varied across provinces and districts). TB = tuberculosis; HIV = human immunodeficiency virus; VITIMES = Viet Nam TB Information Management Electronic System.

METHODS

We retrospectively reviewed data on all TB-HIV patients receiving anti-tuberculosis treatment identified in one of the 24 district TB clinics in HCMC from 1 January to 31 December 2013. The 2013 data were selected to ensure that TB staff at the district level had had time to familiarise themselves with VITIMES, as Phase II was implemented in the districts in 2011.

Data entry and updates for the paper TB treatment register and VITIMES were performed by clinic staff. As soon as a TB diagnosis was determined and treatment started, individual patient data were recorded in the TB treatment register based on individual TB treatment cards and sputum smear microscopy and/or culture results. Updates on patients' bacteriological and clinical responses to TB treatment were entered into the TB treatment register based on the most recent results for smear microscopy, culture and treatment outcomes. Updated HIV-related data such as HIV status, cotrimoxazole preventive therapy (CPT), CD4 cell count and antiretroviral therapy (ART) were provided verbally or in writing by the HIV out-patient clinic where the patient was receiving HIV services. Data entry and updates into VITIMES were usually performed within 1 week of receiving the data.

Individual patient data from VITIMES were exported into a separate analytic database by staff at the provincial TB programme. Only data on TB patients whose HIV status was positive were extracted from VITIMES. Data from the paper TB treatment register were abstracted by district TB staff into an Excel database (Microsoft Corp, Redmond, WA, USA) at the district TB clinics. To assure the quality of the data abstracted, a pre-determined proportion of abstracted records in each district was selected randomly and checked for completeness and accuracy against the paper TB treatment register. At sites with 10 or more records, recheck of another random sample in the district was required if one abstracted record was found to be incomplete or inaccurate.

Patient data in VITIMES and the TB treatment register were matched on full name, age and address. For the purpose of assessing the key characteristics of the individuals with both HIV and TB, we selected 10 of 14 variables recorded in the TB treatment register for evaluation. These indicators included age, sex, enrolment date for TB treatment, TB type (sputum smear-positive pulmonary TB [PTB], sputum smear-negative PTB, extra-pulmonary TB [EPTB]), patient category (new, relapse, failure, retreatment after loss to follow-up [LTFU], other smear-positive, transfer in, other smear-negative/EPTB), TB treatment regimen, TB treatment outcomes (cure, treatment completed, died, failure, LTFU, transfer out, not evaluated), ART (not yet ART, ART before TB diagnosis, ART during TB treatment), CD4 count and CPT (not yet CPT, CPT before TB diagnosis, and CPT during TB treatment). The coding of TB treatment regimens followed WHO guidelines.²

Sensitivity and completeness were defined according to Centers for Disease Control and Prevention (CDC, Atlanta, GA, USA) guidelines for the evaluation of public health surveillance systems.³ Completeness was defined as the proportion of records with completed entries (without unknown or missing entries) and was evaluated for each of the above variables in both the paper and electronic systems. Completeness was reported in two ways: the proportion of completed records among all records entered in VITIMES and the proportion of completed records in VI-TIMES using the denominator of completed cases in the paper register. Sensitivity was calculated as the proportion of TB cases reported to VITIMES compared to the paper system.

The agreement of individual entries in VITIMES with those for linked cases in the TB treatment register was evaluated using κ. As κ works with categorical data, continuous data (age and CD4 count) were converted into categorical data. Only non-missing data were used for agreement calculations. The κ value was interpreted according to Altman⁴ as poor (<0.20), fair (0.21–0.40), moderate (0.41–0.60), good (0.61–0.80) and very good (0.81–1.00).

The protocol underwent formal ethical review by the CDC (Atlanta, GA, USA) and was approved as a routine surveillance activity.

RESULTS

During the period from 1 January to 31 December 2013, a total of 12 485 patients were registered for treatment at one of the 24 HCMC district TB clinics, of whom 12 200 (97.7%) had a value available for HIV status. Of these, 1144 (9.4%) patients registered for TB treatment were identified in the paper-based TB treatment register as being HIV-infected, 1135 of whom had a matching record in VITIMES. The sensitivity of VITIMES compared with the paper register (as the gold standard) was 99.2%. There were no records in VITIMES that could not be matched to a record in the paper register. Of the nine records present in the paper register but missing from VITIMES, one TB clinic accounted for seven missing records, and two other clinics had one missing record each.

Data in the paper register were 100% complete (i.e., all 1144 records had entries) for the following variables: age, sex, enrolment date for TB treatment, TB type, patient category and TB treatment regimen. Among the 1135 records in VITIMES, completeness was also 100% for these variables. Compared with all 1144 cases in the paper register, 1135 (99.2%) cases had data available in VITIMES (Table 1). Completeness in the paper register for TB treatment outcomes and all HIV variables ranged from 91.0% for CD4 count to 99.8% for TB treatment outcome. Of these variables, all 1041 records with CD4 count entries in the paper register also had entries in VITIMES. Among the subset of the 1135 records in VITIMES corresponding to complete paper-based records, completeness in VITIMES was 89.5% for ART status, 93.7% for TB treatment outcomes and 96.6% for CPT.

TABLE 1.

Completeness of records of HIV-infected TB patients with a value recorded for 10 selected variables by VITIMES and the TB treatment register

Agreement of the VITIMES data with the data in the paper registries ranged from 78% to 96%, and were therefore categorised according to Altman⁴ as good and very good agreement (Table 2).

TABLE 2.

Agreement of selected data variables on HIV-infected TB patients in VITIMES with those in the TB treatment register (n = 1135)

DISCUSSION

The WHO encourages TB and HIV programmes to collaborate to ensure that individuals living with TB and HIV receive high quality health care, including early diagnosis and initiation of ART and TB treatment. One component of this collaboration is to ensure timely and high-quality case reporting with strong data linkage between programmes. In accordance with WHO recommendations, in 2008 the Viet Nam NTP modified the national TB treatment register to include the following HIV data fields: HIV testing and status, CD4 cell count, CPT and ART.

Evaluations of the use of electronic TB surveillance systems in many countries have demonstrated varying levels of success.^5–10 Generally, electronic surveillance systems have been slow to be incorporated into routine use and have had some degree of inaccuracy. Our study from HCMC in Viet Nam found a high sensitivity of VITIMES, and high completeness and high agreement of HIV-infected TB patient data from VITIMES with the paper TB treatment register. These achievements may be due to the availability of designated staff for VITIMES data entry at the district level, adherence of TB staff to national guidelines for data entry into VITIMES, regular supervision and technical assistance by the provincial team and good collaboration in data sharing from HIV programmes at provincial and district levels.

Although there was data completeness and agreement, there were incomplete data in both VITIMES and the paper-based register. Incomplete data were noted for indicators that required follow up or sometimes changed over the course of TB treatment and required updating and re-entry, such as TB treatment outcomes, ART, CPT and CD4. Distinct differences in completeness and agreement of VITIMES data against the TB treatment register were found across district TB clinics for some variables. The lowest levels of data completeness and agreement, however, were limited to only a few clinics. Three of the 24 district TB clinics had the lowest data completeness levels, ranging between 48.8% and 56.3%. The remaining clinics demonstrated between 74.3% and 100% completeness for the ART variable. The lowest levels of data agreement (moderate agreement, κ 0.41–0.60) were found in one clinic for the variables age, TB type and TB treatment regimen, in three clinics for the enrolment date variable, in five clinics for the CPT variable and in six clinics for the ART variable. The fact that only a few clinics demonstrated low values for data completeness and agreement suggests that situational or personnel factors (i.e., transcription errors, inadequate training, etc.) may have been responsible, and that targeted site supervision and mentoring at these sites might help improve performance.

Periodic reviews of VITIMES data at the district level are needed to ensure data are complete, consistent and updated. Due to the ongoing nature of anti-tuberculosis treatment, health care providers should ask patients at follow-up visits about changes in HIV status and receipt of services to ensure appropriate TB and HIV care. District staff adherence to the national guidelines on data entry is critical to ensure timely updating and re-entry of data. Furthermore, feedback on the quality and completeness of data in VITIMES at the provincial level should be provided to district TB clinics to enable continuous quality improvement. Our data showed a high sensitivity for TB case detection; HIV variables, however, were found to have some incomplete fields. HIV-related data for individuals with TB, such as ART, CD4 and CPT, are kept by the HIV clinics that provide HIV services, and the data are often not available to TB clinics and are likely under-reported, especially in provinces where there is limited collaboration and a considerable distance between the TB and HIV clinics. The Ministry of Health should promote strategies to facilitate routine data sharing between TB and HIV programmes. As more health programmes transition to electronic systems, VITIMES should be able to link to HIV and other clinical data to allow simplified data sharing and reduce the work burden among TB staff.

Our evaluation of selected variables had some limitations. The findings from this study may not be generalisable to all Viet Nam, given the unique setup of HCMC, where HIV clinics in all districts facilitate data sharing between the district HIV and TB clinics, there are designated staff at district TB clinics for VITIMES data entry and updating and regular site supervisory visits are performed by the provincial VITIMES team. As this study reviewed only HIV-positive TB patients, the results for TB patients without HIV may differ, although the differences are likely to be minimal given that the data entry into VITIMES is almost the same for all TB patients, irrespective of HIV status. In addition, we used the paper TB treatment register as the gold standard in our study. The completeness and accuracy of the TB treatment register itself against other data sources was not checked, and there may have been errors in reporting.

The findings from this study suggest that VITIMES is working well in HCMC and that it can gradually replace the current paper TB treatment register and reports as a potential tool to manage TB cases, monitor TB treatment outcomes and create reliable reports. Efforts are needed to make HIV data more complete.