A Comparison of Electronic Medical Record Data to Paper Records in Antiretroviral Therapy Clinic in Ethiopia: What is affecting the Quality of the Data?

Abstract

Background

Anti-Retroviral Therapy (ART) care is a lifelong treatment, which needs accurate and reliable data collected for long period of time. Poor quality of medical records data remains a challenge and is directly related to the quality of care of patients. To improve this, there is an increasing trend to implement electronic medical record (EMR) in hospitals. However, there is little evidence on the impact of EMR on the quality of health data in low- resource setting hospitals like Ethiopia. This comparative study aims to fill this evidence gap by assessing the completeness and reliability of paper-based and electronic medical records and explore the challenges of ensuring data quality at the Anti-Retroviral Therapy (ART) clinic at the University of Gondar Referral Hospital in Northwest Ethiopia.

Methods

An institution-based comparative cross-sectional study, supplemented with a qualitative approach was conducted from February 1 to March 30, 2017 at the ART clinic of the University of Gondar Hospital. A total of 250 medical records having both electronic and paper-based versions were collected and assessed. A national ART registration form which consists of 40 ART data elements was used as a checklist to assess completeness and reliability dimensions of data quality on medical records of patients on HIV care. Kappa statistics were computed to describe the level of data agreement between paper-based and electronic records across patient characteristics. In-depth interviews were conducted using semi-structured questionnaires with ten key informants to explore the challenges related with the quality of medical records. Responses of the key informant interviews were analyzed using thematic analysis.

Results

The overall completeness of medical records was 78% with 95% CI (70.8% - 85.1%) in paper-based and 76% with 95%CI (67.8% - 83.2%) EMR. The data reliability measured in Kappa statistics shows strong agreements on the socio-demographic data such as educational status 0.93 (0.891, 0.963), WHO staging 0.86 (0.808, 0.906); general appearance 0.83 (0.755, 0.892) and patient referral record 0.87 (0.795, 0.932). The major challenges hindering good data quality was the current side by side dual data documentation practice (the need to document both on the paper and the EMR for a single record), patient overload and low data documentation practice of health workers.

Conclusion

The overall completeness of ART medical records was still slightly better in paper-based records than EMR. The main reason affecting the EMR data quality was the current dual documentation practice both on the paper and electronic for each patient in the hospital and the high load of patients in the clinic. The hospital management need to decide to use either the paper or the electronic system and build the capacity of health workers to improve data quality in the hospital.

Background

Medical records are a combination of both self-reported patient information and physician’s notes on diagnoses, care and treatment. It is used to store and communicate patients’ medical information among health care providers for continuing delivery of medical care [1]. By facilitating the communication among service providers it supports the quality of healthcare [2]. Medical records are also used by government officials for planning, resource allocation, budgeting and other required policy decisions on the health infrastructure [3,4].

Documentation and contents of data within an electronic medical record (EMR) must be accurate, complete, concise, consistent and universally understood by users of the data, and must support the legal business record of the organization by maintaining the required parameters such as consistency, completeness and accuracy [5]. Ensuring high data quality in medical records is fundamental to good clinical practice, program management and ultimately to policy decisions [6].Good quality of data is crucial for patient care and for monitoring the performance of health service delivery [2]. Patients could directly or indirectly benefit from improved data quality, which in turn supports patient satisfaction in care and compliance in their ART program [7].

The quality of documentation in the medical record is dependent upon the consistency and completeness of information entered into the record by all individuals involved in the patient’s care. Increasing interest has been recently demonstrated in the establishment of electronic data systems in resource-limited settings to advance paper-based records [6].However, with the limited resources and capacities, most of the electronic systems are still side by side to the paper documentation which is creating burden on health workers [8].

ART care is a lifelong treatment, which needs accurate and reliable data. As the ART program expands and the number of patients increases, the paper-based medical record continues to be overwhelmingly large and the need for EMR in HIV care clinics is increasing [9]. However, the transformation of the medical record system from paper-based to EMR largely relies on the quality of paper-based system. Hence, improving and ensuring the quality of paper-based medical data is crucial before moving to EMR. It is important to collect data for EMR from paper-based medical records and gradually adopt EMR to simplify the management of large medical records [10]. A comprehensive ART medical record is a cornerstone of good quality and efficient patient care. In subsequent follow-up visits, it can provide a complete and accurate chronology of treatments, patient results and future plans for care [10,11]. Investing in EMR system has dramatically lowered the error rates of HIV care data and has improved client satisfaction [12] as well as quality of HIV care [13].Global funding for the management of HIV has also increased the investment in EMR systems for the management of complex data in monitoring the lifelong treatment of HIV infected people [11]. However, many electronic systems still rely on paper-based medical records for their data inputs and those transition, still practice side by side documentation on the paper and electronic system.

Both paper-based and EMR systems is being practiced because of power fluctuation and inability of hospital to run full EMR systems (14,15,16). Based on generic tools developed by the WHO, paper-based medical records being used in resource-limited settings generally include longitudinal patient flow charts, longitudinal pre-antiretroviral therapy and ART registers, pharmacy registers, appointment books, referral forms and outreach logs that are documented by doctors, nurses, pharmacists, social workers and data clerks [6].Providing and monitoring HIV care and ART requires complete and accurate documentation of patient visit information and laboratory test results [14]. Studies from Kenya, Malawi, Mozambique, Cote d’Ivoire and Rwanda revealed that ART medical records were found to have errors in clinical staging and missing or incomplete data about loss to ART follow-up information [15]. The error rate often increases when inexperienced clinicians are introduced to a new procedure. Studies show that lack of regulations, standards, guidelines, training and education, accreditation, clarity of data elements and data flows as well as health professionals’ and data clerks’ proficiency were determinant factors to the consistency and completeness of ART paper-based and EMR [6,16]. In low- resource settings, there is substantial investment and interest towards computerization but there is limited evidence on the effect of EMRs on the quality of medical records at HIV clinics. Because of this evidence gap, this study aims to assess the completeness and reliability of paper-based and electronic medical records and explore the challenges of ensuring data quality at the Anti-Retroviral Therapy (ART) clinic of the University of Gondar Referral Hospital, Northwest Ethiopia.

Methods

Study Area

The study was conducted at the University of Gondar Referral Hospital, the oldest and biggest referral hospitals serving for more than five million people in North West Ethiopia. The hospital has many specialized clinics, including for chronic illnesses (cardiac disease, renal disease and diabetes), mental health and HIV care. During the study period, a total of 4,907 HIV patients were enrolled on ART at the HIV care clinic of the hospital. Eleven health professionals (nurses, health officers and physicians), four data clerks, six case managers and seven treatment supporters were running the clinic.

Study Design and Period

An institution-based comparative cross-sectional study was conducted to compare completeness and reliability of ART records between paper-based and EMR systems. Both pediatric and adult ART records were used. The quantitative study design was supplemented with a qualitative study to explore the challenges related to the quality of medical records in the clinic. Data were collected from February 1 to March 30, 2017. All patients on ART for at least six months, having complete records both paper-based and electronic (in the local EMR system called SmartCare) and having a follow-up visit in the ART clinic were included in the study.

There were a total of 4,907 ART clients on follow up at the ART clinic of the hospital. Of these, 250 patient records were found to have both electronic and paper-based records. All these 250 patient records were included in the study.

To measure the completeness and reliability of the records, checklists were developed based on the national ART data elements. An overall rate of completeness was calculated by adding up the number of complete data elements per patient visit and dividing it by the total number of expected data elements. This proportion was averaged across all patient visits to get an overall rate of completeness for each visit type. Paper-based medical records and EMR of patients were reviewed to compare the reliability of specific data variables.

Qualitative data was obtained by in-depth interviews of key informants that included physicians, nurses, health officers, data clerks and ART case managers who were purposively selected from the ART clinic. A semi-structured questionnaire was used to conduct the key informant interviews.

Statistical Analysis

Data was entered into EPI info version 7 and then exported to SPSS version 20 for analysis. Descriptive statistics were performed. Chi-square test statistics were conducted to check statistically significant differences between EMR and paper-based medical records’ data completeness. To declare statistical significance a p-value of less than 0.05 was used. Reliability between paper-based and EMR both for the enrolment and follow-up visits were determined using Cohen’s Kappa statistic. The level of agreement was used as defined by Cohen’s kappa(poor agreement < 0.2, fair agreement = 0.2-0.40, moderate agreement =0.41-0.60, good agreement = 0.61-0.80 and very good agreement = 0.81-1.00) [14]. P-value of 5% or less was considered as statistically significant. The qualitative data was transcribed, coded and thematically synthesized using Open Code version 4.02 software. Coding was based on predefined themes namely; organizational, provider and patient related factors. Finally, a narrative analysis was carried out based on the established themes.

Results

Socio-demographic Characteristics

Out of 250 patient records,135 (54%) were from female patients. Nearly one-third, 29(12%), of the patients were from pediatric ART (aged less than 15 years). Half, 51(20%), of the patients included in the study have no formal education and 76(30.4%) of them were from rural residence (Table 1).

Table 1. Distribution of socio-demographic characteristics of medical records of ART patients (n=250), University of Gondar Referral Hospital, Northwest Ethiopia.

Variables		n(%)
Sex	Male	101(40.4)
	Female	135(54)
	Not filled	14(5.6)
Age (years)	1-15	29(11.6)
	16-30	86(34.4)
	31-60	134(53.6)
	Not filled	1(0.4)
Marital status	Never married	51(20.4)
	Married	95(38)
	Divorced	66(26.4)
	Widowed	12(4.8)
	Not filled	26(10.4)
Level of education	No education	51(20.4)
	Primary	66(26.4)
	Secondary	88(35.2)
	Tertiary	32(12.8)
	Not filled	13(5.2)
Occupation	Unemployed	12(4.8)
	Government employee	16(6.4)
	Student	11(4.4)
	Farmer	11(4.4)
	Other	121(48.4)
	Not filled	79(31.6)
Living Area	Urban	171(68.4)
	Rural	76(30.4)
	Not filled	3(1.2)

Completeness of Medical Records

In this study the overall completeness of medical records in the ART clinic was195(78%) with 95%CI (70.8%-85.1%) on paper-based and 189 (76%) with 95%CI (67.8%-83.2%) on EMR, p-value 0.369 (see figure 1).

Figure 1.

Overall completeness of paper-based vs. EMR (n=250), University of Gondar Referral Hospital, Northwest Ethiopia

The average completeness of demographic data elements was 234(93.4%) in paper-based and 232(93.0%) in EMR. Among all data elements, the record completeness of patients’ sex was significantly lower- 236(94.4%), in paper-based than in EMR- 249 (99.6%), (p-value 0.001). Record completeness of patients’ occupation was significantly higher in paper-based than in EMR-(196(78%) vs 171(68%) (p-value 0.001) (Table 2).

Table 2. Completeness of socio-demographic variables in paper-based vs EMR (n=250), University of Gondar Referral Hospital, Northwest Ethiopia.

Data elements	Completeness of Records				P-value
	Paper-based		Electronic
	n	%	n	%
Age	249	99.6	246	98.4	0.130
Sex	236	94.4	249	99.6	0.001
Marital status	224	89.6	226	91.2	0.668
Religion	245	98.0	245	98.0	1.000
Educational status	237	94.8	235	94.0	0.594
Occupation	196	78.4	171	68.4	0.001
Patient Address	247	98.8	250	100	-
Mean	234	93.4	232	93.0	0.625

Regarding clinical data elements, the average completeness was 172 (68.8%) in paper-based and 164 (65.7%) in EMR. Completeness of patients’ weight data was higher with 194(78%) in paper-based than with 175(70%) in EMR (p-value 0.009). Similarly, the completeness of pregnancy status was significantly better with 184 (74%) in paper-based than with 155(62%) in EMR (p-value <0.001). The completeness of WHO clinical staging data was significantly higher in paper-based (77.6) than in EMR (69.6). Data regarding eligibility for ART, reason for referral, and family HIV history had significantly higher level of completeness in EMR than in paper-based medical records (Table 3).

Table 3. Completeness of ART and HIV related variables in paper-based vs EMR(n=250), University of Gondar Referral Hospital, Northwest Ethiopia.

	Completeness of Records				P-Value
Data elements	Paper-based		Electronic
	n	%	n	%
Months on ART	170	68.0	171	68.3	0.892
Weight	194	77.6	175	70.0	0.009
BMI	179	71.6	169	67.5	0.177
Pregnancy status	184	74.0	155	62.0	<0.001
WHO staging	194	77.6	174	69.6	0.006
CPT dose	169	67.7	170	67.8	0.892
ART adherence	116	46.2	129	51.4	0.100
ART regimen	176	70.4	173	69.2	0.681
Functional status	197	78.8	178	71.2	0.008
TB screen	197	78.8	180	72.0	0.017
CPT adherence	195	78.0	205	82.0	0.100
Side effect	191	76.4	177	71.0	0.051
Next visit date	194	77.6	173	69.3	0.004
OI screening	147	58.8	63	25.2	<0.001
Eligibility for ART	135	54.0	186	74.4	0.000
Referral	239	95.6	240	96.0	0.747
Reason for referral	140	56.0	198	79.2	0.000
Disclosure	184	76.3	171	68.4	0.077
Family HIV history	113	45.2	132	52.8	0.016
Prophylaxis’s for OI	118	47.2	66	26.4	<0.001
Mean	172	68.8	164	65.7	0.287

Reliability of Medical Records

Data from a total of 250 ART patients was reviewed for reliability of paper-based and electronic records. Strong agreements were observed on socio-demographic data such as marital status 0.95 (95% CI: 0.912, 0.983); religion 0.93(95% CI: 0.853, 0.986) and educational status0.93 (95% CI: 0.891, 0.963). Similarly, WHO staging 0.86 (95% CI: 0.808, 0.906); general appearance 0.83 (95% CI: 0.755, 0.892) and patient referral 0.87 (95% CI: 0.795, 0.932) had strong agreements. Patients’ address showed a good 0.74 (95% CI: 0.652, 0.828) level of agreement and HIV disclosure to family member had a moderate 0.68 (95% CI: 0.588, 0.759) level of agreement between paper-based and electronic records. The age of the patient had the lowest level of agreement (27.4.

Table 4. Agreement of selected data elements between electronic and paper-based ART records using kappa statistics (n=250), University of Gondar Referral Hospital, Northwest Ethiopia.

Data elements	Kappa coefficient (95% CI)	Percent agreement
Patient age	0.14 (0.093, 0.193)	27.4%
Patient sex	0.87 (0.812, 0.927)***	93.2%
Marital status	0.95 (0.912, 0.983)***	96.4%
Religion	0.93 (0.853, 0.986)***	98%
Educational status	0.93 (0.891, 0.963)***	94.8%
Occupation	0.50 (0.415, 0.569)	56.8%
Patient address	0.74 (0.652, 0.828)**	89.6%
Patient referral	0.87 (0.795, 0.932)***	94.4%
Reason for referral	0.53 (0.433, 0.607)	66.8%
HIV disclosure to family	0.68 (0.588, 0.759)*	80.8%
Family HIV data	0.56 (0.465, 0.652)	73.6%
Patient OI history	0.39 (0.241, 0.426)	64%
Past prophylaxis	0.46 (0.342, 0.540)	69.6%
Pregnancy status	0.59 (0.507, 0.675)	76%
WHO staging	0.86 (0.808, 0.906)***	89.6%
General appearance	0.83 (0.755, 0.892)***	92.8%
Eligibility criteria to ART	0.65 (0.652, 0.741)	78.8%

Strong agreement K***=0.81-1, good agreement K**= 0.60-0.8, moderate agreement K*=0.40-0.60, data variables with true agreement between electronic and paper-based records are explained by the percent agreement.

Kappa statistics was also computed to assess the agreement between paper-based and electronic records through different ART visits. The value of kappa agreement for CD4 count decreases from 0.893 on first visit to 0.205 on the sixth visit. The kappa agreement for ART regimen data increased from 0.672 to 0.939 on first visit and third visit respectively (Figure 2).

Figure 2.

Kappa agreement of selected paper-based and electronic data elements on ART patient medical records (n=250) on follow-up, University of Gondar Referral Hospital, Northwest Ethiopia

Challenges on Improving the Quality of ART Records

Ten key informant interviews were conducted among key informants including five healthcare providers, four data clerks and one case manager. Six of the key informants were females. The median (range)age and work experience of the key informants were 28 years (24-58) and 2 years (range 2-7) respectively. The key informants reported the major challenges hindering the quality of medical record data is the current dual documentation practice and the patient load in the hospital. One of the nurse in the ART clinic said

“We are overwhelmingly busy as we manage more than 100 patients per day. But after handling all this patient, we are expected to write all the patient history on the paper and later into the EMR system which is creating a lot of burden on us”

The attitude of care providers on the need to properly document data is also varied. One ART nurse in the clinic said “Isn’t it enough if we give proper care to the clients instead of wasting our time writing on the paper and the system?”

Updating the data elements as clients came for different visits is also not a usual practice in the hospital. A 29 years old senior physician who has worked for more than 5 years in the ART clinic said that

“ART patient data may not be accurate, for example; BMI data variables recorded during the first visit of the patient are copied to the rest of consecutive visits without updated patient height and weight measurements in the paper-based medical record”.

Additionally, the technical support given to the ART clinic to use the EMR system was reported to be not enough. Head of the ART Clinic said:

“we know that migrating all the data to the EMR will be important. We do not want to stop at 250 but the support we got is very minimal. We need back up power. If power goes, we have to go back to the paper passed records once we have all the data in the computer”.

The overall results of the key informants outline that the current dual documentation practice and the daily patient load are the main challenges for health workers for not having quality data in the ART clinic.

Discussion

This study aimed to determine the completeness and reliability of paper-based medical record and EMR and explore the challenges of ensuring data quality at the ART clinic of the University of Gondar Referral Hospital. This study showed the overall completeness of medical records in the paper-based version was still slightly higher than the EMR. This indicates that there is no enough attention given in the implementation of the EMR in the clinic as the clinic staff are currently documenting both on the paper and electronic systems. It is known that EMRs are implemented in hospitals to improve quality of data, quality of care and the quality of the health service to patients. however, in this clinic, the quality of the data was still better in the paper based system. This may be due to the fact that the ART clinic is using dual documentation systems (paper and electronic). The data quality on the paper finding is consistent with the study from Mozambique [5] which reported 72% completeness in paper-based medical recording and relatively higher than the study conducted in Cote d’Ivoire [11] which reported 62% completeness on ART follow-up.

Data completeness was also assessed specifically for a variety of HIV care data elements on both paper-based and EMR of the ART clinic. Varying data completeness were observed among the specific data elements. For instance, completeness of patient weight data and pregnancy status were higher in paper-based than in EMR. Data elements such as WHO staging, functional status, TB screen, next visit date, opportunistic infections (OIs)screening and prophylaxis’s for OIs have also significantly better level of completeness in paper-based records than in EMR. This result shows that the data on the paper were not properly transcribed in to the EMR. As outlined by another study in Ethiopia [11], this might be due to the workload introduced by the parallel documentation. A previous study in the study area reported that workload significantly affected documentation of the healthcare provided to patients [17].

In both paper-based medical record and EMR, completeness of HIV care and ART data elements was generally lower than the completeness of sociodemographic data elements. This might be due to the nature of the data by which sociodemographic data do not need frequent updates while it is important to update clinical data generated from every visit. Furthermore, sociodemographic data is easier to obtain than clinical data. The difference might also be due to the attitude of care providers to document data [17].

Clinical parameters such as patients’ WHO clinical staging, general appearance, and patient referral have strong agreements. In addition, the consistency of patient address and HIV status disclosure shows a good and moderate level of agreement between paper-based and electronic records. Further analysis on reliability on baseline and follow-up values shows a sharp reduction of reliability from 0.89 on the first visit to 0.21 on the most recent visit. This result reflects that the ART follow-up data elements are not being updated.

The major evidence in this research is the impact of dual documentation on the quality of the data both in the paper based and electronic systems. Different previous evidences [18,19] show dual documentation practice is taking the time of health workers but in this research, we were able to identify that the quality of the data is affected as the result of the dual documentation. Many EMR Implementations in developing countries (22) report dual documentation practice but it will be advisable to either use the paper-based record or to totally transition to Electronic documentation systems to avoid double effort on health workers.

Conclusion

The overall ART data quality was still slightly better in paper-based records than the electronic medical record system. The main reason affecting the EMR data quality was the current dual documentation practice both on the paper and electronic for each patient in the hospital. The hospital management need to decide to use either the paper or the electronic system so that health workers can save time by single documentation practice. Trainings and continuous support to health workers is recommended to build the capacity of health workers on data documentation practices

Limitations

One of the limitation of the study is that the number of data elements used to compare the paper and electronic system is small. To mitigate it, we supplement the study with qualitative data using key informants. More robust study using large sample size and advanced statistical modeling approach is recommended and planned by our team in the future.