Electronic laboratory system reduces errors in National T uberculosis Program: a cluster randomized controlled trial

SUMMARY

OBJECTIVE:

To evaluate the impact of the e-Chasqui laboratory information system in reducing reporting errors compared to the current paper system.

DESIGN:

Cluster randomized controlled trial in 76 health centers (HCs) between 2004 and 2008.

METHODS:

Baseline data were collected every 4 months for 12 months. HCs were then randomly assigned to intervention (e-Chasqui) or control (paper). Further data were collected for the same months the following year. Comparisons were made between intervention and control HCs, and before and after the intervention.

RESULTS:

Intervention HCs had respectively 82% and 87% fewer errors in reporting results for drug susceptibility tests (2.1% vs. 11.9%, P < 0.001, OR 0.17, 95%CI 0.09–0.31) and cultures (2.0% vs. 15.1%, P < 0.001, OR 0.13, 95%CI 0.07–0.24), than control HCs. Preventing missing results through online viewing accounted for at least 72% of all errors. e-Chasqui users sent on average three electronic error reports per week to the laboratories.

CONCLUSIONS:

e-Chasqui reduced the number of missing laboratory results at point-of-care health centers. Clinical users confirmed viewing electronic results not available on paper. Reporting errors to the laboratory using e-Chasqui promoted continuous quality improvement. The e-Chasqui laboratory information system is an important part of laboratory infrastructure improvements to support multidrug-resistant tuberculosis care in Peru.

LABORATORY SERVICES for tuberculosis (TB) are prone to delays in reporting results, as laboratory tests are centralized at regional or national level.¹ Reporting results requires physically moving paper reports from laboratories to the point of care, often including several stops for administrative purposes, which affects the ability to provide effective treatment and prevents programs from tracking performance or estimating total burden of disease.² Similar problems have been reported in the management of human immunodeficiency virus infection.³

Timely and accurate reporting of laboratory and clinical data using electronic systems has reduced the time to administer treatment,⁴ thereby reducing the spread of the disease.⁵ However, no evaluations have been performed in underserved settings.⁶ Timely provision of drug susceptibility test (DST) results should also allow for more appropriate drug regimens, shorter treatment times and improved outcomes.⁷ Lastly, the provision of more accurate data to the drug ordering mechanisms, such as that implemented within the Partners In Health Electronic Medical Record (PIH-EMR),^8,9 should reduce costs and stockouts.

e-Chasqui, a laboratory reporting information system that communicates TB results to clinicians and public health administrators, was implemented in the two largest health districts in Peru in 2006. The system has been expanded to a third district and is currently used by over 200 health centers.¹⁰ It has also been connected to the PIH-EMR,¹¹ which contains clinical data on all multidrug-resistant TB (MDR-TB) and extensively drug-resistant tuberculosis (XDR-TB) cases, which is used by the National TB Program (NTP)¹¹ to facilitate case identification and treatment.

We analyzed a cluster randomized controlled trial (RCT) of e-Chasqui to evaluate its effectiveness in reducing the error rate in reporting laboratory results to clinical personnel in the NTP in Peru.

METHODS

Using a cluster RCT, we tested the effect of the e-Chasqui laboratory information system in reducing errors in communicating test results from district laboratories to health centers. As a secondary study design, we conducted a before-and-after trial. Both trials were performed within an observational study evaluating the impact of expanded laboratory capacity.^1,12

Study settings

The study was performed in two health districts of Lima, Peru. Lima Ciudad includes 45 health establishments serving a population of 1 577 090 in a 100 km² area. Lima Este includes 134 health establishments serving a population of 1 088 515 in a 6340 km² area. Sputum smear microscopy is used to diagnose active pulmonary TB, while culture and DST are reserved for confirmed TB cases with at least one risk factor for MDR-TB, as per NTP recommendations.¹³

Study design

In 2006, e-Chasqui was first implemented in the National Reference Laboratory and two district laboratories of Lima Ciudad and Lima Este. After full implementation in the laboratories, we randomly assigned six health centers (HCs) from each health district (12 in total) to the intervention. In Lima Ciudad, these six were randomized from the 20 highest incidence HCs. Only these 20 high-incident HCs were included in the analysis. In Lima Este, they were randomized from the 12 ‘non-stop reporting’ HCs within the city limits to minimize the difference in distance travelled. ‘One-stop reporting’ HCs (n = 44) belonged to the study arm that was assigned to the HC that transmitted their result (Figure 1). ‘Non-stop reporting HCs’ are in charge of a region; they receive results directly from the regional laboratory and then transmit them to other HCs in their region (one-stop reporting HCs), as seen in Figure 2.

Figure 1.

Study design.

Figure 2.

Flow of samples, results, and start of MDR treatment within the Peruvian National Tuberculosis Program. Smear microscopy is performed at the laboratories in the non-stop HCs and hospitals. One-stop HCs send sputum samples to their closest HC. For patients with MDR-TB risk factors, smear-positive samples are sent to the regional laboratory for culture and/or DST for first-line drugs. Isolates resistant to isoniazid, rifampicin or both are sent to the National Reference Laboratory for confirmation and DST for second-line drugs. Paper results are sent in a step-wise manner to the requesting HC. Numbers of study institutions are in parentheses. DST = drug susceptibility testing; MDR = multidrug-resistant; Tx = treatment; HC = health center; MDR-TB = MDR tuberculosis. This image can be viewed online in color at http://www.ingentaconnect.com/content/iuatld/ijtld/2010/00000014/00000008/art00015

The study was approved by the Partners Healthcare Human Research Committee and the Peruvian National Institute of Health.

Intervention and reporting method

The e-Chasqui web-based laboratory information system was designed and implemented to improve the timeliness and quality of laboratory data in Lima, Peru.¹⁰ The core of e-Chasqui is a patient page that shows the patient’s full bacteriological history, as well as the full result details of the selected sample. Tools built for the laboratory include quality control, reports on tests performed, warnings about delayed reporting of results and a user directory to control access. Tools for clinicians include e-mail notification of new results, a consolidated list of results for their jurisdiction, a list to track the status of all pending samples and a system to send error reports to the laboratories. All intervention HC staff were trained and given access at their HC in an initial visit of approximately 1 h. The data administrator visited or called the HC at least twice a month and could be contacted during business hours. To protect patient confidentiality, e-Chasqui incorporates extensive encryption and web security features and logging of pages viewed, and all users signed a confidentiality agreement.

On accessing the system, personnel can view results and track a sample’s status online. We worked with health districts to change their policy to enable clinical personnel to print out results for treatment start committee meetings. In parallel, the laboratory would send out an official paper result, as per prior routine, that HC staff would place in the patient chart, replacing the printed electronic result. The prior routine was used for the control HCs where the paper results are sent from the regional laboratory non-stop reporting HCs to the one-stop reporting HCs in their region (Figure 2).

Outcomes

An error was defined as when information from the laboratory register did not match the result found in the clinical chart at the HC (paper system) or in e-Chasqui (electronic system). Study staff recorded all relevant variables collected for DSTs and cultures at the HC, including patient name, result date, identification number, result, and if the result or clinical chart was available at the HC. The research coordinator catalogued all errors, and these were verified by one of the investigators. In the case of a discrepancy, a second investigator was consulted. In this paper, we report only major errors: 1) a change in the patient’s name that could result in misidentification, 2) difference in result (e.g., negative instead of positive, incorrect semi-quantitative result) or 3) paper result not found in the patient’s chart (for control HCs) or results not viewed in e-Chasqui (for intervention HCs). If a chart was missing at the HC, the data were excluded. For the comparison of the patient’s name, we considered the chart at the HC to be the gold standard. For the test results, the laboratory register was the gold standard.

We performed a primary analysis comparing the results from the laboratory register with the chart at the HC. If a result was viewed in e-Chasqui, it was counted as not missing. We performed two secondary analyses: first, we analyzed the same data comparing only paper results found at the HC with the laboratory register, without taking into account viewing in e-Chasqui; second, we analyzed only those results that had reached the HC for the number of errors resulting from a wrong name or test result.

Usability and acceptability of the system

A survey previously used in Peru¹⁴ was modified for our intervention and validated with employees from our organizations. After study completion, intervention HC personnel were given the same survey as control HC personnel, with two additional sections for questions about e-Chasqui. Control HC personnel were asked about ‘an electronic laboratory information system’, and not specifically e-Chasqui. The responses were multiple choice, short answers or on a five-point Likert scale anchored by 1 = very positive, 5 = very negative. The survey examined two themes: the frequency of missing results in the paper and e-Chasqui systems, and the security of both systems.

Data abstraction

For cultures, we collected consecutive results every 4 months the year before the intervention and the year after. We used the result date as the inclusion criteria and included all results within those dates. In Lima Ciudad, the pre- and post-implementation collection dates were the first 14 days of March, July and November in 2005 and 2006, respectively. In Lima Este, the pre-implementation collection dates were the first 23 days of May 2005, September 2005 and January 2006, and again 12 months later for post-implementation; more days were sampled because the Lima Este laboratory performs fewer tests than Lima Ciudad.

All DSTs performed pre- and post-implementation in Lima Ciudad (17 January–31 December 2005 and 1 February–31 December 2006) were sampled. No pre-implementation DSTs were performed by Lima Este, as it lacked the required infrastructure or training.

Statistical analysis

We measured data at the sample level, adjusting for the impact on variance of the clusters in the study design. We used a binomial generalized linear mixed model (GLMM)^15,16 with HC as a random effect, patient as a nested random effect within HC due to repeated samples, and pre-intervention mean error rate per HC (as a proxy for HC variance), DST method and number of changes of TB clinicians and nurses per HC as fixed effects. The pre-intervention error rate should adjust for possible HC differences that may have been unequally distributed despite randomization. We stratified the analysis by health district (Lima Ciudad and Lima Este), but as there was no significant difference between them, the stratification is not shown.

RESULTS

Characteristics of the intervention and control HCs are summarized in Table 1. There was no significant difference in the total number of cultures and DSTs between the intervention and control HCs. There was a significantly higher number of clinician changes and a lower baseline culture error rate in the intervention HCs, although the baseline DST error rate and all other characteristics were similar: 98% of all culture results and 100% of all DST results available in e-Chasqui were viewed by the intervention HCs. Respectively 5.7% and 9.0% of patient charts could not be located in the control and intervention HCs and were not included in the analysis (Figure 3).

Table 1.

Characteristics for all study HCs

Characteristic	Control HCs mean (SD)	Intervention HCs mean (SD)	P value
Total ‘non-stop reporting’ HCs, n	20	12
Total ‘one-stop reporting’ HCs, n	27	17
Monthly cultures per HC	12.3 (11.1)	26.3 (28.6)	0.06
Monthly DST per HC	2.0 (2.0)	3.4 (3.8)	0.06
Changes in TB clinician per HC during study	1.7 (1.1)	2.1 (1.1)	0.04
Changes in TB nurse per HC during study	1.4 (0.9)	1.6 (0.9)	0.34
Pre-intervention error rate for cultures, %	15.9	11.0	0.04
Pre-intervention error rate for DST, %	18.6	23.6	0.15
Sample sizes, n
Cultures sampled	498	697
DST sampled	561	709
	Before	After
Cultures sampled	1448	1195
DST sampled	304	712

HC = health center; SD = standard deviation; DST = drug susceptibility test; TB = tuberculosis.

Figure 3.

Flow of samples through trial. HC = health center; DST = drug susceptibility test.

Primary analysis

For DSTs, the intervention HCs had 82% fewer errors than control HCs (2.1% vs. 11.9%, P < 0.001). For cultures, the intervention HCs had 87% fewer (2.0% vs. 15.1%, P < 0.001; Table 2).

Table 2.

Primary and secondary analysis for DST and cultures in the randomized controlled trial. Samples had at least one error

Outcome	Control HCs %	Intervention HCs %	Odds ratio (95%CI)	P value
Primary analysis
DSTs	11.9	2.1	0.17 (0.09–0.31)	<0.001
Cultures	15.1	2.0	0.13 (0.07–0.24)	<0.001
Paper charts only*
DSTs	11.9	12.3	1.11 (0.76–1.62)	0.60
Cultures	15.1	11.9	0.84 (0.57–1.23)	0.37
Subset of results found at HCs
DSTs	2.0	2.1	1.10 (0.46–2.62)	0.82
Cultures	1.8	2.0	1.15 (0.47–2.82)	0.76

^*Not including online viewing.

DST = drug susceptibility test; HC = health center; CI = confidence interval.

Secondary analysis

Looking at the paper results, the major source of errors was missing results, which accounted for 72– 86% of all errors depending on the comparison. When comparing the paper system with the printout from the electronic system placed in the patient record, there was no significant difference in the e rror rate for either DST (P = 0.60) or cultures (P = 0.37) between the control and intervention HCs (Table 2).

In the second analysis, we took the sub-group of results with only ‘incorrect name’ or ‘incorrect result’ errors, excluding missing results. No difference was seen between control and intervention HCs for DSTs or cultures (Table 2). After implementing e-Chasqui, however, there was a 60% decrease for DST (P = 0.01) and 50% for cultures (P = 0.008).

Usability and acceptability of the system

Twenty-three users did not participate in the survey because they were not present at their HC. The response rate was respectively 94% (29 of 31) and 93% (108/116) in the intervention and control HCs (Table 3). The intervention HC users had, on average, more years of internet usage and had accessed the internet more frequently from the TB office or laboratory. These can be at least partly attributed to e-Chasqui, as the district office had prioritized internet access for these HCs.

Table 3.

Survey respondent characteristics

Characteristic	Controls (n = 108) n (%)	Intervention (n = 29) n (%)
Sex
Female	81 (75)	17 (59)
Male	27 (25)	12 (41)
Clinical background
Physician	20 (19)	6 (21)
Nurse	35 (32)	8 (28)
Laboratory staff	9 (8)	3 (10)
Nurse technician	36 (33)	10 (34)
Other	5 (5)	2 (7)
No response	3 (3)	0
Internet usage, years
0	26 (25)	2 (7)
1–5	47 (44)	18 (62)
6–10	25 (24)	6 (21)
>10	1 (1)	1 (3)
No response	9 (8)	2 (7)
Location of internet usage*
Tuberculosis office/laboratory	16 (15)	15 (52)
Office in health center	17 (16)	4 (14)
House	56 (53)	12 (41)
Internet cabin	36 (34)	7 (24)
Other office	17 (16)	3 (10)
Other	3 (3)	0
Health district
Lima Ciudad	89 (82)	17 (59)
Lima Este	19 (18)	12 (41)

^*More than one response was possible for this section.

At least 55% of users reported not receiving at least 10% of results through the paper system alone; 73% of users felt this reduced the opportunity for treatment (Table 4). All of the users in the inter vention HCs found results in e-Chasqui that they did not have on paper.

Table 4.

Users’ opinion of the paper and e-Chasqui systems^*

	Controls (n = 108) n (%)	Intervention (n = 29) n (%)
How often were you missing a culture or DST result for a patient? (paper system)
1 of 2 patients	27 (26)	2 (8)
1 of 4	21 (21)	3 (12)
1 of 10	22 (22)	11 (44)
1 of 50	13 (13)	2 (8)
1 of 100	2 (2)	0
Never	17 (17)	7 (28)
Do you believe this diminished the opportunity for treatment?
Yes	80 (80)	16 (73)
No	20 (20)	6 (27)
Did you find information in e-Chasqui that you would not have had without the system?
Yes		20 (69)
No		9 (31)
In which system do you believe the information is more complete (the requests are filled out better)?
Electronic/e-Chasqui	83 (82)	21 (75)
Paper	0	1 (4)
Both are the same	18 (18)	6 (21)
In which system is the information more confidential (accessible only to the appropriate personnel)?
Electronic/e-Chasqui	77 (75)	27 (96)
Paper	4 (4)	0
Both are the same	21 (21)	1 (4)
In which system are the data more secure (will not be lost)?
Electronic/e-Chasqui	84 (84)	26 (93)
Paper	4 (4)	0
Both are the same	12 (12)	2 (7)

^*Percentages are calculated from total number of responses to the question. Percentages may not equal 100 due to rounding.

DST = drug susceptibility test.

Error reporting

There was an increase in the number of electronic ‘error reports’ submitted by HC staff to the laboratory using e-Chasqui. The mean number of error reports per month was respectively 8.3, 39.2 and 60.4 in 2007, 2008 and 2009.

DISCUSSION

This RCT showed that intervention HCs had a reduction in errors of respectively 82% and 87% compared with control HCs for DSTs and cultures. This suggests that access to e-Chasqui had a significant impact in reducing the error rate, mostly by providing access to results that were otherwise unavailable on paper. These missing results accounted for at least 72% of all errors. This error rate may underestimate the impact of e-Chasqui, as errors due to missing clinical charts were excluded. These would constitute an error in the paper system, but not in the electronic system, as results are permanently available.

There was no difference in the proportion of paper results missing from the intervention HCs compared to the control HCs. This means that although results were viewed electronically, personnel in the intervention HCs were not printing them if the paper result had not arrived, possibly due to a lack of printing capabilities or viewing outside the HC.

Error rates were significantly lower in the intervention HCs after the implementation, suggesting a significant improvement due to the laboratory use of e-Chasqui. The reduction in the misspelling of names and incorrect reporting of results affected the entire health district, as the paper results for both control and intervention HCs were printed from e-Chasqui, although only the intervention HCs had electronic access to the system.

In surveys, the majority of all users reported that they were missing at least 1 in 10 results, validating our study results. More importantly, almost 70% of e-Chasqui users found results electronically that had been missing on paper. This may be the system’s largest impact.

We have found few reviews of data quality in other resource-poor settings,³ but our experience leads us to believe that quality issues are of the same or greater magnitude than described here (R Hurtado, personal communication, 2006),^17,18 and information systems such as e-Chasqui could have similar benefits in these settings. A high rate of missing results can easily occur in a paper system. This is aggravated by many factors in public health care systems, including high patient load, lack of administrative staff, inconsistent transportation of samples and results, and lack of storage space or organization for charts.

The impact of this system could be expected to be larger in less connected or rural settings as the previously described challenges worsen. The resources required to develop and implement this system are described elsewhere,¹⁰ but to extend it to rural Peru, for example, at least intermittent internet connectivity would be required. This could be done by having clinical personnel use internet cabins¹⁹ in their town, if providing the HC with internet is not feasible. Similar tools are being built in open source systems such as OpenMRS (www.openmrs.org), which can be downloaded from the internet.

The study had some limitations. The data were collected at approximately the same time for both pre- and post-implementation phases, requiring pre-implementation data to be stored longer at the HCs. The study was conducted in high-burden HCs in two populous health districts in Peru, and therefore may not be generalizable to other sites. Finally, this was a formative evaluation, as the developers of e-Chasqui were involved.

CONCLUSION

An electronic laboratory reporting system implemented in the Peruvian public health care system significantly reduced the number of missing laboratory results at point-of-care health care sites via electronic viewing. Although the rate of missing results or errors on paper was not affected, the before-and-after comparison showed a significant improvement due to its implementation at the laboratory. Clinical users confirmed that the system provided them with results not available via the paper system, and a further study is being conducted on its clinical impact. Finally, clinical staff acted as quality control agents by providing electronic error reports directly to the laboratories. Such systems should be part of the laboratory infrastructure to support TB and MDR-TB care in resource-poor settings.