Is transcription of data on antiretroviral treatment from electronic to paper-based registers reliable in Malawi?

Abstract

Setting:

Antiretroviral treatment (ART) clinics at one central hospital, three district hospitals and one mission hospital in the central and southern regions of Malawi.

Objective:

To measure the extent of inaccuracies in the transcription of case registration and recorded deaths between electronic medical data (EMR) and paper registers. This was done to inform the Ministry of Health on the reliability of the paper-based system as backup in case of EMR failure.

Design:

Retrospective analysis of routine programme data.

Results:

A total of 31 763 registrations and 2922 deaths in the EMR were compared with those in the paper registers. In one hospital, up to 24% of overall case registrations were missing from the paper registers. At other sites, the differences were minor and included duplicate patients who should have been classified as ‘transfer in’ patients in the paper register. There were major differences in the number of registered deaths in two of the five facilities.

Conclusion:

There are varying degrees of agreement between the EMR and paper registers which compromise the use of the latter as a backup solution in case of EMR failure. The reasons for this unreliability and ways forward to address the problem are discussed.

Faced with a serious human immunodeficiency virus (HIV) epidemic, Malawi has been scaling up its antiretroviral treatment (ART) services since 2004. By the end of June 2010, there were 371 ART clinics and over 300 000 patients had been placed on ART in the country.¹ At the end of each quarter, ART clinics across the country compile quarterly cohort reports on case registration and ART programme outcomes, which are submitted to the Ministry of Health (MoH). With an ever-growing number of patients initiated on ART, the compilation and verification of the quarterly cohort reports has become cumbersome and time consuming.²

To facilitate ART monitoring and reporting, the Malawi MoH decided to move towards a point-of-care electronic medical record (EMR) system.³ Implementation of the system started in 2006. The EMR system was derived from the ART treatment card⁴ and based on specifications from the MoH. The system generates automated quarterly reports on case registrations and patient outcomes, and the data are used by the MoH to compile quarterly national ART reports.

The MoH requires that ART sites that have adopted the EMR system also continue to run paper-based systems to serve as backup in the eventuality of EMR system failures. EMR data are first backed up on the server at the health facility where the system is running and at an off-site central server on a daily basis. Case registration and information on ART outcomes generated through printouts of master card data in the form of adhesive labels from the EMR are manually transcribed by clinic staff into the paper-based register. The ART supervision team observed that there were differences in the data that were transferred and reported using the paper-based system as compared to the EMR. As national reporting and drug forecasting depends on the EMR, inaccuracies in the paper-based system in case of EMR failure could have important programmatic implications, such as incorrect drug forecasting.

Our objective was to assess the extent of inaccuracies in the transcription of case registration and recorded deaths between the EMR and the paper-based data system.

METHODS

Design, study setting and population

We conducted a retrospective audit of routine programme data at five ART sites in the central and southern regions of Malawi. These ART sites included three district hospitals (Dedza, Ntcheu and Salima), one mission hospital (St Gabriel’s Mission Hospital) and one central hospital (Queen Elizabeth Central Hospital [QECH]). The district hospitals each have two nurses, a clinician and two clerks. They typically attend to more than 200 patients on a clinic day, and their ART registers currently have more than 3000 patients each. QECH has four nurses, three clinical officers and two clerks, who attend to more than 400 patients per day; their ART register has more than 10 000 patients. St Gabriel’s Mission Hospital has three nurses, two clinicians and one clerk, who see more than 100 patients a day; their ART register includes more than 200 patients. These sites were chosen as they were among the first sites to start ART in Malawi, had high case loads and were among the first to implement the EMR. All these five sites also run paper-based register systems.

All patients enrolled in the ART programme up to 31 December 2010 at these five ART sites were included in the study. A review of the paper registers was conducted at all five health facilities between January and February 2011.

Information flow procedures at the ART sites

The clinic staff are fully responsible for managing all data related to the patient master cards, the ART register and the EMR. The sequence in terms of use of the two systems is as follows: the clinic staff first fill in the EMR, which automatically generates a visit summary in the form of adhesive label printouts that are then affixed to individual patient master cards. The ART register is supposed to be updated manually at the end of each calendar quarter from the master cards. This allows a comparison of data between the EMR and the paper register.

Using the EMR, real-time data are collected by the health care provider at the point of care. This was thus used as the gold standard for comparison with the paper register. An automated quarterly EMR report is generated which is submitted to the MoH team during their ART quarterly supervision visits.

Data collection, variables and analysis

The overall numbers of patients ever registered for ART and outcome data on reported deaths were independently collected from the EMR and the paper system and compared. Death was used as a proxy to examine possible inaccuracies in transferring treatment outcome data from the EMR to the paper register, as this outcome requires active, manual update of the ART register. Data were entered into a Spreadsheet (Ubuntu Open Office Spreadsheet, Redwood Shores, CA, USA). Differences in count data for overall case registration and death in the two systems were calculated and expressed in percentages with 95% confidence intervals.

Ethics approval

Approval for conducting this research was granted by the National Health Science Research Council, Malawi, and the Ethics Advisory Group of the International Union Against Tuberculosis and Lung Disease, Paris, France.

RESULTS

The numbers of patients registered in the EMR and the paper-based system are shown in Table 1. A total of 31 763 patients were registered in the EMRs, while only 28 359 (89%) were found in the paper register. Major differences in overall numbers of registered patients were observed at QECH. At the other sites, differences were minor and included duplicate patients in the paper register who should have been classified as ‘transfer in’ patients. There were also additional patients in the paper register due to ART dispensary records that were missed in the EMR.

TABLE 1.

Numbers of patients registered in the EMR system and paper registers in five clinics in central and southern Malawi as of 31 December 2010

Site	EMR	Paper register	Difference n (%)*	95%CI
QECH	14 529	11 109	3420 (23.54)	22.86–24.24
Ntcheu	5 521	5 524	−3 (0.05)†	0.02–0.15
St Gabriel	2 815	2 816	−1 (0.04)†	0.01–0.21
Dedza	3 844	3 853	−9 (0.23)†	0.12–0.44
Salima	5 056	5 057	−1 (0.02)†	0–0.11
Total	31 763	28 359

^*Proportional difference = (EMR-paper register/EMR) × 100.

^†Transfer-in patients recorded in paper registers and not in the EMR.

EMR = electronic medical record; CI = confidence interval; QECH = Queen Elizabeth Central Hospital.

The numbers of reported deaths in the EMR and paper-based registers are shown in Table 2. Of 2922 deaths, 2518 (86%) were included in the paper register. There were major differences in death outcomes at two sites (QECH and Ntcheu).

TABLE 2.

Numbers of reported deaths in the EMR system and paper registers in five clinics in central and southern Malawias of 31 December 2010

Site	EMR	Paper register	Difference n (%)*	95%CI
QECH	774	489	285 (37)	33–40
Ntcheu	277	203	74 (27)	22–32
St Gabriel	386	376	10 (3)	1–5
Dedza	749	732	17 (2)	1–4
Salima	736	718	18 (3)	2–4
Total	2922	2518	404 (14)

^*Proportional difference = [(EMR-paper register/EMR)] × 100.

EMR = electronic medical record; CI = confidence interval; QECH = Queen Elizabeth Central Hospital.

DISCUSSION

This is the first study to assess the variability of data transcription from EMR to paper registers at high-burden ART sites in Malawi. There was considerable variability across the ART sites.

This study is important as it assessed the robustness of paper registers in case of EMR failure; this information is useful to the Malawi MoH as well as other countries in sub-Saharan Africa that are either using or planning to use EMR systems. Current EMR implementations in developing countries rely on entering information retrospectively into the EMR from paper-based forms.⁴ This study sheds light on EMR use at point of care.

The strengths of this study are that a large number of patients were included from different high-burden ART sites. As the data came directly from health facilities at the point of care, they are likely to reflect the operational reality on the ground. The sites are supervised by MoH teams on a quarterly basis and are thus robust, and rigorous comparisons of data in the EMR system and paper registers were performed. The study also followed the recommended international STROBE (strengthening the reporting of observational studies in epidemiology) guidelines.⁵

A limitation of the study is that outcomes other than death were not assessed. This would have allowed assessment of inaccuracies of outcomes of patients who were alive, and could be used to estimate the utilisation of drugs.

The findings of this study highlight a number of important observations. First, overall, data on numbers of registered cases were acceptable in four of five ART sites, where differences were minimal. However, at one of the sites (QECH) there was an underreporting of 24%, which is unacceptable, as it rendered register data unreliable as a backup in case of EMR failure. The direct programmatic implications of missing data on, for example, drug forecasting would result in drug stockouts for up to 3500 patients. Such an eventuality could have major repercussions on ART treatment as well as the credibility of the ART programme in the eyes of patients and the community at large.⁶ One of the reasons for the wide difference in register data at the QECH is that paper registers were poorly maintained and as a result one register was reportedly lost and others lacked several pages of register data. As this is a very high-burden centre, workload-related factors might also be to blame.²

Second, small negative differences in overall case registrations in four clinics (other than QECH) were attributed to duplicate recording of patients in the paper register. These were ‘transfer in’ patients who were wrongly assigned a new number in the register instead of maintaining their unique registration number allocated at the time of ART initiation.

Third, although there were minimal differences in data on deaths in three of the five sites, two sites (QECH and Ntcheu) had major differences of respectively 37% and 27%. Differences in outcomes such as deaths are more sensitive indicators of the prompt transcription of data between the EMR, patient cards and registers. Shortcomings at two sites indicate that there is a problem in this regard. Such differences in adverse outcomes would portray the false impression that the ART programme is functioning better than it actually is.

Finally, the EMR was introduced to eventually reduce the workload related to managing large cohorts over time, and reporting on their outcomes.³ Maintaining the current practice of running the two systems along with manual transcription of data from the EMR to the paper register is cumbersome and defeats this objective. Furthermore, as the clinics and the MoH are fully reliant on the EMR system, clinic staff already tend to consider paper-based registers redundant. This might be particularly important where workload is high and human resources are limited.² There is thus a real risk that the reliability of the paper register system will drop further over time. Possible solutions to this problem include 1) the MoH deciding on focusing its energy and resources on the EMR if it is considered to be robust and reliable, and 2) providing regular printouts of updated register data in the same format as paper registers and storing these as backups. This will require equipment, stationery and storage facilities, but would ensure perfect matching of EMR and backups. This will also complement the independent electronic backup of the EMR system and avoid manual data transcription.

In ART clinics in central and southern Malawi, there are limitations in the quality of the transcription of data from the EMR to paper registers which are used as backup, and solutions to address this problem are urgently needed.