Development, implementation and preliminary study of a PDA-based bacteriology collection system

Abstract

Partners In Health (PIH) and its sister organization in Lima, Peru, Socios En Salud (SES), treat a majority of multidrug-resistant tuberculosis (MDR-TB) patients in Peru, in conjunction with the Peruvian National TB Program (NTP). Monthly bacteriology tests, which must be collected from health establishments located across this major city, are an integral part of this treatment. Currently, a SES employee visits each health establishment to collect this information by hand, process it and type it into an electronic medical record system (PIH-EMR).

In this paper, we describe the development and implementation of a personal digital assistant (PDA)-based electronic system to collect, verify and upload monthly bacteriology data into the PIH-EMR. After an initial implementation period, we performed a pilot study to test the use of this system. We completed a baseline assessment in two health districts and then implemented the electronic system in one of the districts while the control site continued to use the paper-based system during the same period. The PDA-based system had a processing time of 6.2 days, significantly lower than measurements for both the baseline [54.8] and control sites [64.4] (both p<0.0001). It was also able to reduce the frequency of discrepancy from 10.1% to 2.8% (p<0.0001) and receive positive feedback from the users. Finally, the system’s cost would be recuperated in three months from time savings due to increased work efficiency. This system will be the subject of a larger study to determine its impact on delays, errors and costs.

Introduction

According to the World Health Organization (WHO), tuberculosis is second only to AIDS as the most deadly infectious disease in the world. Incidence of tuberculosis (TB) in Peru is the highest in South America¹ and in the densely populated periphery of metropolitan Lima the risk of infection with M. tuberculosis is estimated to be among the highest levels documented recently in any population^2–4. Multidrug-resistant tuberculosis (MDR-TB) is now recognized as one of the most significant emerging infectious diseases and jeopardizes even the most highly advanced nations.

The complicated nature of MDR-TB treatment provided the impetus to design a web-based electronic medical record system (PIH-EMR)⁵ to manage the care of MDR-TB patients, model their medication requirements, provide clinical tools, and allow easy information-sharing between multi-site-based physicians and staff. It provides many functions including: (1) an electronic patient registry to maintain patient information for over 5300 patients that currently or previously received treatment; (2) a web-based nurse order entry system for TB medications that has been shown to significantly drop the error rate and improve user satisfaction⁶; (3) a method for electronically maintaining chest radiographs. Further, a previous pilot project provided a PDA to a single nurse to view patient records in the EMR system with the aim of giving healthcare workers low-cost, portable access to such data⁷.

Bacteriology data comprise one of the most important clinical measures of treatment response. Each MDR-TB patient should leave a monthly sample, usually from sputum, at their local health center. The two bacteriology tests currently performed in Peru are smear and culture. Reducing delays in reporting these tests are important because continued positive bacteriologies indicate that the patient is at further risk of lung damage and also increase the possibility of transmission to others. Timeliness of these data is a significant problem in the clinical monitoring of these patients and was the initial impetus for the system described here.

To enable clinical and programmatic monitoring of MDR-TB, SES/PIH collects and stores these data for all of their patients in Peru. In Lima, this includes over 3,500 monthly results from more than 120 health establishments in five health districts distributed over 1030 square miles (2672 sq. km.). In the current method of data collection, a team of four SES employees visit all of the health establishments and record test results by hand on a paper sheet. This sheet is then brought to the SES office, where it is copied onto two other forms and then typed into the PIH-EMR.

The major disadvantages to this paper-based method are the delay times in entering a record into the PIH-EMR, data quality issues, and the work load involved in the process. Assessment of all samples collected in 2004–2005 showed that the average was 48 days from the result date to the PIH-EMR entry date. This is mostly due to the backlog of data that the team members themselves needed to type into the PIH-EMR. In assessing data quality, an internal review previous to this study showed the error rate to be 7.9%. Finally, each bacteriology team member spends approximately half his/her time performing verifications and transcriptions; this could be virtually eliminated with an electronic system.

Materials and Methods

To decrease delay time and errors, we designed and implemented an electronic bacteriology collection system using a PDA as the initial point of data entry at the clinical site. The information is uploaded to the web-based PIH-EMR, where additional pages were created to automate the validation of the data, generation of the required forms, and data transfer into the PIH-EMR.

Hardware and Software Selection

In selecting handheld computers, we compared Palm OS-based systems and Pocket PCs. We chose the low-end Palm OS-based systems (Zire 31 and 21) due to their lower cost, smaller size and monochrome screens. In the poor areas where the bacteriology team collects their information, discretion is important. The Palms’ smaller size made them easier to disguise within a notebook carried by the user and the monochrome screen called less attention.

In selecting software to use, we wanted to be able to do rapid prototyping of forms and be able to connect to the Oracle® database of the PIH-EMR. For these reasons we chose Pendragon® Forms, a commercial application that applies a modified client/server model to the PDA/PC relationship. It is based on a Microsoft Access® database on the PC and has the ability to “hotsync” to any Open Database Connectivity (ODBC)-compliant database. This system allowed us to quickly create forms for entering bacteriology data, as well as to download patient information and to upload completed bacteriology results to the web-based PIH-EMR using Microsoft Access® ODBC connection over the Internet.

During the entire trial period, there have been no incidents of loss, theft or damage to the devices. The users are protective of the PDAs and are aware of their surroundings. In case of theft, all access to data on the PDA has been password protected, thus providing more security for the information than when it was carried on paper sheets.

This bacteriology collection system was designed, developed, and tested with a team experienced in collecting bacteriology results from all over Lima, Peru. At every step of development, we considered the current workflow and the role that an electronic system with decision support could have. We discussed options with the users of the system and found their initial apprehension turned into excitement when the benefits of this system became clear.

We surveyed the work flow performed by the Socios en Salud bacteriology team and found two models of collecting information: (1) Single site model, where a member of the bacteriology team is able to collect both the smear and culture information for a specimen from one regional laboratory; (2) Dual site model, where a member of the team collects the primary smear information from a local health center. He/she then visits the regional laboratory and collects the culture information and a second copy of the smear information. This secondary smear information is cross-checked with the primary smear information obtained at the health center to ensure correct communication between the institutions.

New pages were designed in the PIH-EMR to allow the bacteriology team to process this information before transferring to the bacteriology section of the PIH-EMR for clinical use. They were created taking into account the workflow of the bacteriology team of data collection, verification of data, printing of additional forms and finally entry into the PIH-EMR. The pages were designed to decrease the time required to verify and enter data into the EMR as well as to improve data quality through decision support.

Initial web pages in the PIH-EMR allow users to view all of the information in a table format identical to their previous forms. We found that these users, with low to moderate computer experience, preferred this view because (1) it allowed them to see all of the information at once and (2) it was in a familiar format. The next web page performs data quality checks previously done by the team (Figure 2). They include:

Figure 2.

Data Quality page

1. Not allowing duplicate entry of a test result

2. Reporting any missing data for a specimen (first error in Figure 2)

3. In the dual site model, verifying that the primary and secondary smear information is identical

4. Checking that every culture has a corresponding smear and every smear a corresponding culture (second error in Figure 2)

5. Alerting for any overdue smear or culture that had not been transferred

6. Informing if a patient from a dual site institution is missing a smear from their health center

7. Color coding smears in the two site model to display if the smear information has been collected from both sites and cross-checked

Subsequently, the users print selected results on a standard layout and then transfer all the results to the bacteriology section of the PIH-EMR for clinical and administrative use. These web pages were color-coded depending on the test performed, whether the information had been cross-checked and whether there were remaining errors. If the data quality page detects an error or missing information in the result, it is displayed in gray and cannot be transferred.

Pilot Study

To evaluate the initial impact of this system in data collection we performed a pilot study. After taking baseline measurements in two health districts for four months, we implemented the PDA-based system in one of them (Callao) while the second health district (Lima Ciudad) continued to use the paper-based system. There were two team members in each of the health districts. Thus two PDA were distributed for the intervention health district. For the larger study, the number of PDA users has been expanded to four. After a two month implementation period, we performed a second measurement for a four month period. We then compared the intervention data against the baseline measurement in the same health district (Callao) and against the control health district (Lima Ciudad). The system was also implemented in a third, unrelated health district. However, it is not included in this study because of the bacteriology team has had bureaucratic difficulties in getting access to the information.

The specific aims of the study were as follows:

1. To compare the average processing time using the electronic system to the paper-based system; (processing times is defined as the number of days from collection of bacteriology results at the health center to their entry into the PIH-EMR)

2. To compare the frequency of discrepancies in the bacteriology results entered with and without the electronic system by comparing the information entered into the PIH-EMR to the original laboratory notebook;

3. To assess the system’s usability and acceptability by the users of the system

4. To assess the cost-effectiveness of the electronic system, compared to the current paper forms and conventional data entry.

To compare the average processing time, we calculated mean processing times per sample for all groups and compared the differences in means using a Wilcoxon signed ranked test. To compare discrepancies, we compared the number of samples with and without errors for the two groups using the Fishers exact test. Since there was a close collaboration in developing the system, its usability was gauged by conversations and meeting throughout the development and study. Finally, the initial cost-effectiveness only includes cost of the equipment for the electronic system and the amount of person hours spent on specific tasks before and after its implementation. A more comprehensive analysis will be performed in the full study.

Results

The average processing times were reduced significantly by the PDA-based system in comparing the trial with the baseline and the control data.

The above table shows the average (±standard deviation) per sample for the two health district at the different time measurements. The sample sizes for each time period were between 532 samples (historical control, Callao March–Jun 2005) and 874 (intervention, Callao Sept–Nov 2005). The 54.8 day delay from the baseline measurement fell significantly in the intervention group to 6.2 days per sample (p<0.0001). This same delay was also significantly lower than the 64.4 days in the concurrent control group (Lima Ciudad), p<0.0001. The delay in the control group did not differ statistically from the baseline measurement.

In comparing the frequency of discrepancies we only performed before and after comparison in the intervention group (Lima Callao). This ensured that the user was the same between the intervention and control group. We logged three types of errors: (1) errors of commission where a value for that sample found in the PIH-EMR was different from the laboratory notebook, (2) errors of omission, where a value for that sample was not found in the PIH-EMR but was in the laboratory notebook (3) misidentification errors where a result or a sample were assigned to the wrong patient.

In the paper-based system, we checked 175 smears and 170 cultures against their original laboratory record and found that 35 (10.1%) had at least one error of commission or omission. An initial quality check of the electronic system of 200 smears and 200 cultures revealed 11 (2.8%) samples with at least one error of commission or omission, a rate significantly lower than the conventional system (p<0.0001). No mis-identification errors were found in either group.

Usability

The user response to the electronic system has been positive though the team was initially apprehensive about its use. After 2–5 days, each of the users became comfortable with using the handheld to enter information and found that approximately the same time was required to enter information into the PDA as in the paper system. However, their most favorable response came from being able to quickly verify and transfer the bacteriology results electronically instead of having to work with large amounts of papers. Because of their experience the four users have asked to expand the system to all five health districts in Lima as soon as possible.

Cost effectiveness

To gauge the feasibility of implementing this in all of Peru and in other projects, we provide the initial costs and savings of this electronic system. We measured the amount of staff time needed to process smears and cultures from the initial collection of the information at the health center through entering that information into the EMR. In the initial measurements, using the PDA-based system reduced the time required for the entire process by 25%. The average monthly salary of a team member is USD$250.00 or USD$3000 annually, (higher than the average salary in Peru of USD$2620⁸). Therefore this system provides important initial savings of USD$3000 annually for the entire team.

The cost of a Palm Zire 21 PDA is USD$150.00 and the form software USD$100.00 per PDA, though the prices decreases as more are implemented. Accessories for each PDA, such as extra stylus and protective cover, were USD$15 per PDA. We don’t include the costs of a computer with internet here because we assume that users will already be using one, as was the case for the bacteriology team. Therefore, the total hardware and software costs per PDA were USD$265.00. Further, this system can be expanded to include all health districts in Peru by adding additional PDAs with no other changes needing to be made.

Training for the use of this system consisted of the development of a user guide and four training sessions with the users of approximately four hours each. These sessions included training the team on the use of the PDA and web pages, as well as feedback to improve the system. The training time for new users should be considerably less. Further, there was frequent email and text messaging contact between the developer and users during the entire development and implementation periods. The frequency of contact decreased during the study period, once the system had been completed.

Discussion

Many organizations, especially in developing countries, must collect information about specific populations from a wide area. Handheld systems offer an advantage in being a portable method to digitize information at the initial point of contact and initial experiences have begun to show in what circumstances they are of benefit^{7, 9–11}. In developed countries, it has been shown that in clinical settings, handhelds have the potential to increase communication and reduce the quantity of discrepancies^12–15. However, there are few studies of handheld implementation in developing countries, and almost none of their impact in these locations where the potential for improvement is much larger than in developed countries.

More importantly, this system is undergoing a larger controlled study this year to investigate its effect in reducing delay times, errors and costs, as well as its ability to improve the experience of the users. This study will be finished by August 2006 and should provide more conclusive data on benefits and costs.

Conclusions

This PDA-based laboratory result collection system allows users to gather results from many distinct health establishments and later synchronize it to a central database, to verify all the information and to transfer it to an electronic medical record system. A pilot study showed that this system can significantly decrease delays in processing and errors with a positive user experience. This system’s total cost, including two PDAs and software, is less than $600. With the increased efficiency seen in the pilot study, it should pay for itself within three months.

Our hope is the development of our PDA-based system will help others create their own systems. We have come into contact with many organizations and researchers with similar requirements of collecting data from multiple sources in different locations. We feel that one of the key factors to success was the careful study of workflow, and the close relationship between the developers and end-users.

Figure 1.

PDA form example

Table 1.

Average processing time in days per specimen with standard deviation * and † p<0.0001

Date/DISA	Callao (days)	Lima Ciudad (days)
March–Jun. 2005	54.8±64.1*	55.7±27.6
Sept.–Nov. 2005	6.2±5.1*†	64.4±43.2†