Evaluating the impact and costs of deploying an electronic medical record system to support TB treatment in Peru

Abstract

The PIH-EMR is a Web based electronic medical record that has been in operation for over four years in Peru supporting the treatment of drug resistant TB. We describe here the types of evaluations that have been performed on the EMR to assess its impact on patient care, reporting, logistics and observational research. Formal studies have been performed on components for drug order entry, drug requirements prediction tools and the use of PDAs to collect bacteriology data. In addition less formal data on the use of the EMR for reporting and research are reviewed. Experience and insights from porting the PIH-EMR to the Philippines, and modifying it to support HIV treatment in Haiti and Rwanda are discussed. We propose that additional data of this sort is valuable in assessing medical information systems especially in resource poor areas.

Introduction

As medical information systems and electronic medical records (EMR) become more central to healthcare in the US and other developed countries, such systems are starting to be used in developing countries. This trend has increased markedly in the last 3–5 years as large projects have been set up to treat HIV, Multi-drug resistant tuberculosis (MDR-TB) and Malaria. Funding for these projects, which comes from the Global Fund for HIV, TB and Malaria, the US PEPFAR program and others, has supported the development of a range of information systems in Africa, Latin America and the Caribbean. These system range from spreadsheets and simple databases built by team members, to commercial software (which may be modified for the new purpose), to sophisticated, custom built software. They are used for a wide variety of purposes from clinical care and telemedicine to reporting, accounting and research. There are many challenges to implementing such systems in resource poor areas ranging from lack of stable electricity and communication systems to basic problems of heat, humidity and physical security. Additional to hardware and software costs there are important human costs in terms of training and of the amount of time required by critical staff to maintain the data in the system. This has lead to “push back” from some projects in impoverished areas who see IT in general, and medical records in particular, as luxuries they cannot afford. To address these issues it is necessary to evaluate the use of systems in resource poor areas and assess the benefits and costs of their implementation and use. We describe evaluation studies we have carried out on a large custom built Web-based EMR system for supporting the treatment of MDR-TB in Peru. We emphasize the practical issues of using the system and its impact on care delivery and the work of the staff in Peru, and discuss the limitations of such studies and the barriers to carrying them out. We also review other data that can help assess the impact of the system based on its use in reporting and in research studies, and lessons learned from migrating it to other countries.

Background

The rapid growth of medical records and other medical information systems in the US in recent years has been driven by a belief that such systems can improve the management of healthcare and in particular can improve quality of care and reduce medical errors. These beliefs are based on practical success stories, but also on formal evaluation studies. Evidence of reduced medication errors¹ or better screening and follow-up of diseases² has driven implementation of medical records in many hospitals with a particular emphasis on those incorporating decision support and order entry systems. Successfully implementing such systems is not a simple task. Even the much touted order entry systems may not improve quality of care in all cases³. Staff implementing systems need to carefully study the existing workflow and ensure that the EMR system does not disrupt it in unintended and potentially harmful ways⁴ especially since workflow and safety checks are usually much less robust in resource poor settings. Developing countries offer many additional challenges to system design due to a lack of basic infrastructure and local technical support (reviewed elsewhere⁵). Other very important challenges include lack of familiarity of staff with IT systems especially for health care, and frequently a desperate shortage of trained healthcare staff including doctors, nurses and laboratory technicians.

PIH-EMR in Peru

Peru has the highest incidence of TB in South America and over the last decade around 3% of patients were found to have the more dangerous and difficult to treat MDR-TB. The non-profit organization Partners In Health (PIH) and its sister organization in Peru Socios En Salud (SES) developed an innovative program to treat MDR-TB in Lima in 1996. In 2000 PIH and SES received a grant from the Bill and Melinda Gates Foundation to scale up treatment to all of Peru. Treatment of MDR-TB requires complex drug regimens and close monitoring of bacteriology results for 2 years or more. In 2001 we developed and deployed a web based medical record system, (the PIH-EMR⁶), to assist in the clinical management of these patients. The system collects data on patients’ clinical condition, drug regimens and laboratory results, and includes a set of tools to assist in the detection and correction of data entry errors. The PIH-EMR was designed to support multiple functions including immediate access to clinical information, drug supply logistics, data for clinical research, and programmatic reports for governments and funders. A particular focus of our work is to develop and deploy tools to detect potential omissions or errors in patient management. The EMR also includes more specialized tools to support activities such as psychiatric review and TB surgery. Over 2500 digital images of chest x-rays are accessible allowing monitoring of radiological progression when the original films are not available. The system is implemented in Spanish and English. In the last year, the PIH-EMR had over 32,000 logins by 55 SES staff members. Their roles range from administration and reporting, to clinical care to drug management and logistics. Internet access is relatively cheap in Peru, and the EMR can function over slow dialup connections, or in Haiti with only intermittent connectivity⁷.

Assessing the value of an information system

If we are to move forward in implementing successful information systems in countries such as Peru or Haiti we need to evaluate existing medical record systems in resource poor areas. Good evaluations are difficult to perform and are not common even in developed countries such as the US⁸. Such studies face additional challenges in resource poor areas. These can be practical such as availability of research staff and resources to pay for the study. In addition there are major research ethics concerns in sites where staff are often stretched just to provide care, and time diverted to research can be the “last straw”. Studies need to be carried out with the minimum disruption to the normal work flow, and study results should be of prompt benefit to the project and to the care of our patients. In evaluating the PIH-EMR we set out to determine whether the system was genuinely helpful to the staff rather than just novel, and what functions (if any) clearly contributed to the delivery of care. These constraints lead us to carry out several formal controlled evaluations of different component of the PIH-EMR. In addition we have looked at other data on the usability of the system for reporting and to support observational research. Finally we looked at the implications of using the EMR in other countries.

Strategies for evaluation

Stead et al⁹ have proposed a five-stage framework for the evaluation of medical informatics projects; (1) problem definition, (2) bench testing in the laboratory, (3) early field trials under the direct control of the original investigator, (4) field testing in new or unfamiliar settings, (5) definitive study of the system’s efficacy during routine operational use. Stages 1 and 2 are about good design and implementation of the software and ensuring that the code performs correctly to that design. Stage 3 is the key initial phase of clinical testing of the system. Surprisingly few systems undergo such tests, but even when they do many questions can be left unanswered. Is this system still used after the intensive early phase of implementation and training? Can the developers be objective about their creation? Are data complete and accurate? In addition few software systems work well beyond the original development site. Successfully implementing a system in a new site run by a different organization and staff (stage 4) is an important test of the potential long term impact. The goal of Stage 5 should normally be a randomized, controlled trial of the stable system in a new environment, unfortunately a rare event.

More specific study questions include:

Does the system improve health care delivery?

• Accuracy, timeliness or completeness of data

• Access to data in locations or times that were not previously possible

• Communication and/or collaboration

• Monitoring or analysis of care

• More effective scale up of patient numbers

• Reduction of medical errors

• Improvement in quality of care – the most important but challenging question

What are the downsides of using such a system in a developing country?

• Cost of system development including hardware, networking, software and system configuration

• Cost of training, and introducing new software

• Ongoing cost of data entry and management

• Possible misuse of information

Alternative costs - what would have to be done or improved anyway?

• Reporting to funders and health ministries

• Tracking and forecasting drug supply

• Performing quality control on data collected in treatment sites

• Backing up data in case of loss or subsequent editing of records

• Setting up communications in remote sites

Designs and Results of Evaluation Studies

Evaluations of the PIH-EMR include:

1. a series of small structured evaluations

2. assessment of analysis and reporting functions

3. assessment of research support

4. assessment of implementation in new sites

1) Formal evaluation studies

A Web-based Nurse Order Entry System for MDR-TB Patients in Peru

This study¹⁰ was designed to assess the effect of introducing a new set of forms for drug order entries by the nurses and their assistants running the treatment program. The new system replaced one based on paper forms and Excel spreadsheets that were emailed to the pharmacy. This required the same data to be entered electronically three times (into the spreadsheet, into the EMR and into the pharmacy system for dispensing). With the new order entry system data are entered once into the EMR and transmitted to the pharmacy or printed for archiving in paper charts. Decision support tools are incorporated to warn of problems with drug doses, interactions or a lab result showing drug resistance.

The system was studied by assessing error rates and delays in data entry in two health districts at baseline. We compared drug regimen data from the electronic medical record, paper charts and pharmacy records. After a two-month training period on the new system, we conducted the trial for 52 days in two of Lima’s six health districts, and re-reviewed the medication data. There was no significant difference in the control group’s error rate before or after the study (8.6% vs. 6.9%, P=0.66). The intervention group, however, showed a significant drop in the error rate (17.4% vs. 3.1%, P=0.0074). Additionally, the nurse expressed satisfaction with the order entry system and its ease of use. However the study was small involving four staff in the two groups. After these promising results we started to implement the system in all the districts in the treatment project. Progress was slow due in part to many small but important changes required to ensure all drug regimens could be entered easily. As a result staff maintained parallel systems (old and new) for over a year in some districts. Now we have successfully switched all districts to the order entry system which is working well.

Tools to assess drug requirements

The PIH-EMR includes analysis tools to calculate the drug requirements for groups of patients by totaling all the drug regimens in the EMR. These tools are used to assess future drug requirements and reduce the risk of stock-outs. Comparing the prediction of drug usage in 2003 with the record for drugs actually dispensed from the pharmacy showed less than a 3% discrepancy between the estimates¹¹. We have subsequently shown that the discrepancy is less than 1% for years 2002 and 2003 in total. Recent work has demonstrated that future drug requirements can be predicted up to a year in advance. As medication costs in 2003 were $2.7 million (63% of total treatment costs) effective procurement and supply tools have the potential to generate large savings.

Use of a Palm Pilot based data collection tool for bacteriology data

Bacteriology test results are crucial to tracking the progress and outcome of MDR-TB treatment but in Lima these data come from over 120 laboratories and health centers. A team of four staff must visit all the sites weekly and copy results into notebooks for later entry into the EMR. We developed a Palm Pilot based tool for bacteriology data collection, automatic upload to the EMR and data cleaning. We evaluated data in one district before and after implementation for 16 weeks and also compared it to a control district with the old system¹². Mean delay in entering results in the EMR was 6.2 days in the intervention site versus 54.8 and 64.4 days for the baseline and control (both P<0.0001). Mean errors were also reduced to 2.8% from 10.1% (P<0.0001). It took the staff 25% less time to collect and enter the data, representing potential savings of about $3000 per year. Hardware and software costs were $600 in addition to the time required for training and setup. A definitive controlled study of the system is currently in progress.

2) Analysis and reporting

The EMR is used weekly to generate custom reports internally, and for funding agencies and the Peruvian health ministry. We have also developed a suite of tools to assess data quality and detect delays, omissions or errors. Data managers make wide use these tools for early recognition of data problems, and frequently ask for refinements and additions. These tasks would require substantial additional time and effort without the EMR, some like finding duplicate records would be nearly impossible in large datasets.

3) Research studies

The PIH-EMR has been used to support a wide variety of research studies on the treatment of MDR-TB. Data from the EMR has been used for over a dozen abstracts and 9 papers, with several others in press.

The advantages of using the system for research are:

• The technical and security infrastructure is already in place and does not need to be created

• A single master patient list with ID numbers is constantly maintained and updated

• Data entry occurs in near real time which can simplify the process of tracking down missing or erroneous results

• Questions and hypotheses can be rapidly tested against the data using Web based analysis tools

• The EMR has a well tested temporal data model designed for following patients with chronic diseases which allows time series data analyses

• Research data collection and cleaning can improve the quality of the clinical records

Disadvantages

• Some forms with simple data structure can be slower to implement in the EMR compared to a basic database

• Researchers often prefer a simple flat-file data model to simplify analysis and may require extra training or additional data extraction tools

• Designing the EMR around research studies may make it more complex for clinical care.

4) Implementing the PIH-EMR in new sites

IT projects are challenging to implement beyond their original site. The PIH-EMR has been ported to three other countries supporting TB and HIV treatment.

Philippines

In October 2004 the PIH-EMR was set up at the Tropical Disease Foundation in Manila, the Philippines to support an MDR-TB treatment project there. The code was upgraded to include a configuration file for switching the appearance of the forms and for the geographical localization required for that project. Three of the authors (HF, DJ and SC) traveled to Manila to assess the workflow, finalize the forms and train the staff. The local installation is called the TDF-EMR and runs on a server in the clinic, but the code base is identical for the Peru and Philippines projects. To date the TDF-EMR has records on 1279 patients who have started or completed treatment including 489 with MDR-TB, and clinic staff login around 60 times per day on average.

Haiti

In 2002 the PIH-EMR system was modified to support the care of HIV in developing countries. This included adding new forms and converting the language choices to French and English for use in our clinics in rural Haiti. A new off-line component was created that allows the entry and viewing of patient records when the internet is temporarily unavailable. The HIV-EMR⁷ now has records on over 7000 HIV positive patients including 2300 who are on anti-retroviral treatment. It is used for clinical care and monitoring of quality of care, as well as reporting to funders and for observational research.

Rwanda

In 2005 the HIV-EMR system from Haiti was set up in Rwanda to support the treatment of HIV patients in 6 clinics there. The system was modified to handle differences in forms and workflow such as the generation of summaries of treatment groups before follow up visits. New tools were added to allow local staff to create government reports. After 10 months of operation there are 2000 patients in the system of whom 1300 are on ARV therapy.

Discussion and recommendations

The evaluations of the PIH-EMR have allowed us to better understand how such systems can function in resource poor environments. Many lessons are common to all environments; careful workflow analysis and focus on the most important needs, excellent and meticulous software design, good training, and systems for follow-up and support. Training and supporting local leadership and data managers is critical to long term success. Studies described here have helped to explain to local staff why certain functions are important and their involvement in running studies or as subjects has usually had a positive effect on system use. On the other hand nurses, doctors and pharmacy staff tend to be very conservative about changes in their working practices and changing the system for drug regimen entry took long term education, encouragement and support. It is likely that the time saving from avoiding duplicate data entry finally won them over. Tools for predicting drug requirements are now in regular use and evaluation studies have been crucial to their acceptance. Larger studies including tools for prediction of drug requirements a year in advance are nearing completion. The drug prediction tools are also in use in the Philippines and Haiti. The Palm Pilot system has proved very popular from the start and even before the evaluation is complete the staff are requesting its use in new districts. We expect that handheld devices will prove useful in other projects, with the proviso that our bacteriology collection system was very carefully designed and engineered for a specific, high value task.

The successful porting of the PIH-EMR to the Philippines, and its modification to support HIV treatment, suggests that it fulfills a more general need for medical information management. Many functions are in use in all sites, but a significant number of changes and additions were required to be successful. Conversely some software components for reporting and monitoring of quality of care, developed for Haiti or Rwanda, are now being adopted in Peru. Setting up the PIH-EMR in the Philippines cost approximately $45,000 for programming, travel and training. Costs would be lower for another site as modifications were made to improve flexibility and configurability of the EMR. We are developing a new architecture for EMR systems in developing countries called OpenMRS¹³ in collaboration with colleagues at the Regenstrief Institute in Indiana, Moi University, Kenya and University of KwaZulu-Natal, South Africa. The OpenMRS is more configurable for new projects, diseases, countries and languages.

Many questions about the use and value of the PIH-EMR remain however. How well can it function entirely independent of the development team? The Philippines site comes closest to this situation but we are still providing technical support and some programming of additional functionality. How important are functions that Web-based systems can provide like better communication of results between sites? Can the PIH-EMR improve quality of care? The latter question is being addressed in a new study of rapid communication of drug sensitivity test results between laboratories and clinics in Lima (stage 4 of the schema by Stead⁹). A new component of the PIH-EMR called “e-Chasqui” is being evaluated for its ability to shorten the time it takes for failing TB patients to be switched to effective treatment for their drug resistant TB. The cost and management of this study is being shared with a study of rapid test methods for MDR-TB supported by the US Centers for Disease Control and Prevention and lead by Drs. Sonya Shin (PIH) and Peter Cegielski (CDC).

Conclusions

The PIH-EMR has now been in use for almost five years in Peru and 2 years in the Philippines. The HIV-EMR version has been in use for three years in Haiti and a year in Rwanda. Total records for patients in all sites exceed 20,000 with more than 7000 currently receiving treatment. Initial evaluations have demonstrated that the system is widely used. In addition several components are associated with reductions in data errors, delays in data entry, and workload compared to the existing paper and spreadsheet approaches. All four systems are heavily used for reporting to funders and governments as well as for internal management. Despite these promising results we need to carry out more definitive evaluations of the stable system, to determine if the considerable cost and energy involved in setting up an EMR is justified by long lasting improvements in patient care and reductions in work load for essential tasks.