Data quality in electronic medical records in Manitoba

Abstract

Objective

To determine if the problem list (health conditions) in primary care electronic medical records (EMRs) accurately reflects the conditions for which chronic medications are prescribed in the EMR.

Design

A retrospective analysis of EMR data.

Setting

Eighteen primary care clinics across rural and urban Manitoba using the Accuro EMR.

Participants

Data from the EMRs of active patients seen in an 18-month period (December 18, 2011, to June 18, 2013, or December 3, 2012, to June 3, 2014) were used.

Main outcome measures

The likelihood of documentation in the EMR problem list of those specific chronic diseases for which drug prescriptions were documented in the EMR. Regression modeling was performed to determine the effect of clinic patient load and remuneration type on the completeness of EMR problem lists.

Results

Overall problem-list completeness was low but was highest for diabetes and lowest for insomnia. Fee-for-service clinics generally had lower problem-list completeness than salaried clinics did for all prescription medications examined. Panel size did not affect problem-list completeness rates.

Conclusion

The low EMR problem-list completeness suggests that this field is not reliable for use in quality improvement initiatives or research until higher reliability has been demonstrated. Further research is recommended to explore the reasons for the poor quality and to support improvement efforts.

As electronic medical record (EMR) systems have been adopted in hospital and family practice settings across Canada, their potential benefits for disease surveillance and management have been noted.¹ Studies have demonstrated that EMRs can support physicians in the delivery of primary care services, including faster access to patient information, collaboration between care providers, decision support at the point of care, improved efficiency by avoiding duplications, and better information management.^2–7 Despite increased implementation of EMRs in Canada,⁸ it has been previously reported that only 14% of Canadian physicians are using their EMRs in a clinically significant way.^9,10

Electronic medical record systems have become the focus of many studies globally, covering a range of topics including implementation, the effect on patient care and work flow, and data quality. Although the number of Canadian studies lags behind those performed in other countries, several have begun to emerge.^11–13 Notably, investigators at the eHealth Observatory at the University of Victoria in British Columbia have developed a framework to assess EMR-based health information, laboratory management, referrals, and decision support.¹⁰ This tool has been used to study EMR adoption in Manitoba.¹⁴ While EMR adoption is important, it only captures part of the story, so there is a need to explore data quality to better understand how EMRs are being used.

With the development of national EMR networks for health surveillance, such as the Canadian Primary Care Sentinel Surveillance Network,³ EMR databases have also been recognized as an invaluable source of health information for research purposes. However, high-quality data entry and completeness of the data are required in order to use these data to reliably measure primary care activities and health outcomes. In particular, aspects of completeness, concordance, consistency, and correctness must be verified. Previous Canadian studies have demonstrated low data quality in the Canadian context,¹⁵ but research in this area is limited. There is a need to continue examining the validity of data based on information extracted from EMRs in order to determine its reliability and suitability for use in research and surveillance. In this study, we examined whether the problem list (health conditions) within primary care EMRs accurately reflects the types of prescriptions written for patients with chronic disease.

METHODS

We compared the diagnoses listed in the EMR problem list with prescriptions written for specific chronic diseases, including hyperlipidemia, gout, chronic obstructive pulmonary disorder (COPD), hypothyroidism, diabetes, osteoporosis or Paget disease, and Alzheimer disease or dementia. We also included 2 common diagnoses that have treatments with solitary indications (ie, insomnia and migraines).

Participating clinics

Primary care clinics were recruited based on their use of the Accuro EMR for at least 18 months. We included salaried (ie, alternative or block funding) and independent fee-for-service primary care clinics in order to determine whether the quality of problem lists was affected by method of payment (fee for service compared with salaried) or panel size. We also included a subgroup of salaried clinics that had a focus on family medicine resident training. We termed those clinics residency training clinics, as most practices have 1 or more residents participating in the provision of care per primary care provider. Consent was obtained from primary care providers at each clinic and resulted in access to 18 clinics and 118 practitioners with approximately 64 787 active patients. Ethics approval for this study was received from the Winnipeg Regional Health Authority (WRHA) and the University of Manitoba Health Research Ethics Board.

Data collection

Data were collected retrospectively from both urban and rural primary care clinics in Manitoba that were using QHR Technology’s Accuro EMR. The first phase of data collection was conducted in the WRHA, which has a shared EMR database. Within WRHA clinics, data were collected for active patients seen between December 18, 2011, and June 18, 2013. For non-WRHA clinics, data were collected through remote or direct access to individual clinic EMRs and queried in a similar manner; data were collected from December 3, 2012, to June 3, 2014. Data were extracted from fields detailing the clinic office, care provider, prescription medication, and problem-list diagnosis code. Only structured data that used the ICD-9 (International Classification of Diseases) coding or ATC (Anatomical Therapeutic Chemical) system were captured for problem-list diagnoses and prescriptions.

The data were obtained from the EMRs using queries designed based on the University of Victoria’s eHealth Observatory EMR Data Quality: Evaluation Guide.¹¹ These queries were designed to monitor only active patients, who were defined as having an “active” status in the EMR and who had also received care from the providers within an 18-month time period.¹⁶

We defined EMR completeness as the number of patients who had received a prescription for a specific chronic disease and who also had that disease documented in the EMR’s problem list. Table 1 defines the matching medications and diagnoses. Medications were chosen that had only a single indication to ensure the correct medication diagnosis pairing.

Table 1.

Medications identified and their associated conditions

DRUGS	DISEASE
Statins	Hyperlipidemia
Allopurinol	Gout
Tiotropium	Chronic obstructive pulmonary disorder
Levothyroxine	Hypothyroidism
Hypoglycemic agents (metformin excluded)	Diabetes
Zopiclone	Insomnia
Bisphosphonates	Osteoporosis and Paget disease
Triptans	Migraine
Donepezil	Alzheimer disease and dementia

We defined the number of patients in a practice as the number of individual patients seen at least once by a primary care provider at the participating clinic’s practice in the 18-month period of data extraction. Full-time equivalent (FTE) hours were calculated as the amount of time each physician spent providing clinical care in a typical week. We assigned a value of 0.1 per half-day for 1 primary care provider, meaning a full-time 5-day work week equaled 1.0. We collected this information by directly contacting the clinic managers. The FTE hours were totaled per clinic, as many of the clinics in our study had a moderate or large amount of cross-coverage relating to patient care. A panel size was determined for each clinic and calculated as the total number of active patients per clinic divided by the total FTE hours worked per clinic.

Analyses

Data completeness was determined by the number of patients who had EMR documentation of a disease divided by the total number of patients who had a prescription documented for a medication that would normally be prescribed for management of that particular disease. Mean completeness values were calculated for each disease. The effect of the various characteristics was calculated by determining the ratio of the covariance parameter estimate and its standard error, where a ratio greater than 2 is generally an estimate of significance.

We used modeling that placed the data in a natural multi-level structure, with patients nested within physicians and physicians in turn nested within clinics. Consequently, generalized linear mixed models were used to allow for correlation between patients when estimating the relationship between clinic type, patient load, and completeness, and to quantify the extent to which physicians and clinics differed in completeness rates. These models extend traditional logistic regression to include random effects arising from multi-level data. Completeness was considered binary and was treated as the dependent variable. Odds ratios and their confidence intervals are reported. The PROC GLIMMIX add-on for SAS, version 9.3, was used for all analyses.

RESULTS

The practices included in our analysis are described in Table 2.

Table 2.

Summary of participating clinic practice information

CLINIC TYPE	TOTAL NO. OF PATIENTS	NO. OF CLINICS	NO. OF PHYSICIANS	AVERAGE FTE HOURS*	AVERAGE PANEL SIZE†
Fee for service
• Rural	18 716	2	23	6.3	1475
• Urban	10 955	2	11	4.0	1327
Salaried
• Rural	5414	5	11	1.7	683
• Urban	19 009	6	48	5.8	555
Salaried residency training clinics
• Urban	10 693	3	25	4.3	799

FTE—full-time equivalent.

^*The FTE hours were totaled per clinic, as many of the clinics in this study had a moderate or large amount of cross-coverage relating to patient care in the 18 months studied. The FTE hours were calculated as the amount of time each physician spent providing clinical care in a typical week. A value of 0.1 per half-day for 1 primary care provider was assigned, meaning a full-time 5-day work week equaled 1.0.

^†Panel size was determined for each clinic and calculated by the following formula: total no. of active patients per clinic divided by the total FTE hours worked per clinic.

Data completeness for all clinics combined was first determined by calculating the mean sensitivity of the problem list for each diagnosis. We found that completeness was highest for diabetes and lowest for insomnia (Figure 1). Fee-for-service clinics overall were found to be associated with lower EMR problem-list completeness compared with the salaried and residency training clinics (Table 3). Similarly, residency training clinics generally had lower problem-list completeness than other salaried clinics (Table 3). The statistical linear mixed models that were used allowed for the calculation of odds ratios, which demonstrate the likelihood that a condition would be found in the problem list, but did not allow for the calculation of specific P values for each prescription-diagnosis combination.

Figure 1.

Mean sensitivity of the problem list for each chronic disease: Error bars represent the standard error of the mean.

COPD—chronic obstructive pulmonary disease.

^*Excluding metformin.

Table 3.

Odds ratios demonstrating the effect of salary type and panel size on EMR problem-list completeness

MEDICATION	ODDS RATIOS FOR PROBLEM-LIST COMPLETENESS
	PANEL SIZE	FFS VS SALARIED	FFS VS RESIDENCY TRAINING	SALARIED VS RESIDENCY TRAINING
Statins	1.0950	0.149	0.531	3.558
Allopurinol	1.0797	0.206	0.342	1.660
Tiotropium	1.3097	0.587	0.663	1.129
Levothyroxine	1.0575	0.132	0.280	2.116
Hypoglycemic agents	1.2584	0.060	0.383	6.390
Zopiclone	1.1356	0.297	0.749	2.519
Bisphosphonates	1.0704	0.093	0.253	2.723
Triptans	1.1103	0.151	0.238	1.576
Donepezil	1.2738	0.197	0.777	3.939

EMR—electronic medical record, FFS—fee for service.

Table 3 also demonstrates that we did not find a correlation between the number of patients per primary care clinician (panel size) and EMR problem-list completeness. We considered this factor based on clinic averages for FTEs, as the clinics in our study had some level of shared care among clinicians.

Interestingly, we found that wide variability in EMR completeness existed at the physician level, and this was highly associated with overall clinic EMR problem-list completeness for each of the 7 chronic diseases. Table 4 illustrates this variability, as average rates of physician-level completeness are widely different for each chronic disease, and the differences between the 25th and 75th percentile for each disease are substantial. Problem-list completeness for COPD, as well as for Alzheimer disease or dementia, showed the highest variability at the physician level, while insomnia and diabetes showed the lowest variability at the physician level.

Table 4.

Physician completeness rates for different diseases

DISEASE	PHYSICIAN COMPLETENESS RATES
	AVERAGE RATE, %	INTERQUARTILE RANGE, %
Hyperlipidemia	43	39
Gout	51	50
Chronic obstructive pulmonary disorder	60	75
Hypothyroidism	65	36
Diabetes	79	31
Insomnia	20	27
Osteoporosis and Paget disease	51	51
Migraine	50	55
Alzheimer disease and dementia	64	67

DISCUSSION

To our knowledge, this is the most comprehensive Canadian study involving EMR data quality related to prescribing in a primary care clinic setting. Recently, EMR completeness was also investigated in an Ontario-based study comparing primary care data with administrative data (mostly inpatient), in which analyses were performed on many aspects of patient care involving prescriptions, physician visits, laboratory tests, hospital discharge information, and billing.¹⁷ The authors concluded that overall, EMR and administrative data compared fairly well, although specific areas for improvement were identified, such as better integration between hospital and nonhospital records. Compared with this study from Ontario and other studies measuring problem-list completeness in the United Kingdom, we found that EMR completeness in Manitoba was lower.¹⁸ Overall, problem-list completeness for most diseases in our study ranged between 40% and 60%, except for diabetes, which had an 82% completeness rate. In addition to that, Hassey et al reported sensitivities for hypertension, diabetes, hypothyroidism, asthma, and coronary artery disease of 97.8%, 98.3%, 82.1%, 87.3%, and 95.5%, respectively,¹⁸ and Faulconer and de Lusignan reported sensitivity for COPD of 79%¹⁹; thus their problem-list completeness was considerably higher than that in our study.

There is a variability in the literature regarding the particular measures used to assess problem-list accuracy and this has previously been pointed out as a limitation in the United Kingdom.¹³ The specific measures used in this study are similar, although not identical, to those used in other jurisdictions. Caution must be applied when attempting to directly compare the mean completeness among different methods of testing. However, the high variability in sensitivity among providers in this study, as demonstrated by the width of interquartile ranges (Table 4), shows that even within this study population, problem-list completeness varied highly between providers. This underlies how strong an effect the clinician providing and documenting the care has on EMR data quality.

Further analysis by clinic type identified lower completeness at fee-for-service clinics compared with the salaried clinics for all diseases (Table 3). There are many possible explanations for this finding in terms of the way care is delivered in these environments. The most striking difference in the Manitoba setting is the presence of an interdisciplinary team in salaried and residency training clinics as opposed to fee-for-service practices. Both the residency training and salaried clinics offer a range of coexisting services by multiple nonphysician clinicians such as primary care nurses, dietitians, and social workers. It is possible that team-based care encourages providers to be more diligent with problem-list completeness or that providers other than physicians (eg, nurses) might be maintaining and contributing to this part of the EMR. Currently, we are unable to access user logs to determine whether this is the case. An alternative explanation is that for fee-for-service physicians “time is money,” so they do not always take the time to keep problem lists up to date.

With regard to the poorer results seen in residency training clinics compared with salaried clinics, there might be an effect related to the maturity and experience of the provider inputting information in the problem list. In clinics primarily devoted to residency training, the bulk of patient care is provided by learners. Novice clinicians might be more focused on factors related to patient care that do not involve maintaining an accurate problem list, even though it could be argued that the need for accurate problem lists might be higher in this environment, where physician continuity of care is compromised by the need to provide the residents with clinical experience.

Previous studies have identified panel size as a factor contributing to the overall quality of patient care,^20,21 and we hypothesized that panel size could influence problem-list completeness. In our study, however, no correlation was found between clinic panel size and EMR problem-list completeness.

Our explanations regarding the cause of the data quality differences are speculative, as our study was not designed to determine the reasons for these differences. However, qualitative work by Ivers et al²² showed that physicians are often focused on immediate patient needs rather than the management of more complex chronic disease cohorts. The value of keeping an updated problem list in relation to what is being prescribed for a patient was not consistent based on our findings. There are many uses for a problem list, including provision of care in cases where the provider is not intimately familiar with the patient’s history and use of the data for secondary purposes such as quality improvement or research. If these are not seen as priorities at the point of care, then the use of the EMR for anything beyond the “here and now” might be limited. This might be exacerbated in fee-for-service environments, where volume of patient flow has a direct effect on income. The low data quality in our study indicates that this might be true in all environments, with an exacerbating influence of the funding model and practice type.

Certain diseases in our study were much more likely to be present in the problem list. This could relate to the acuity of the condition and the need for chronic management with laboratory testing, as seen with diabetes and hypothyroidism. It might also relate to the perceived importance. While not captured in our statistical analysis, 2 of the authors (A.S. and S.Y.) presented the raw data to the participating clinics and were offered many insights into why the problem lists were incomplete in many cases. For the use of statins and sedatives to treat insomnia, clinicians indicated that they did not feel compelled to include the specific indication for the prescription in the problem list. In part, this might be because the diagnosis is captured within another diagnosis (eg, previous myocardial infarction for statins).

Nevertheless, these data are important as they underscore the variability in primary care providers’ use of EMRs specifically as it relates to coding diagnoses in problem lists. This is an emerging area of concern, as EMR data will be increasingly used for purposes not related solely to immediate patient care.

Limitations

We were limited in the types of diseases we could use for this analysis because we focused on diseases that had a specific medication (or class) with a specific indication. This methodology might have overestimated the accuracy of diabetes, for example, owing to the exclusion of metformin, which is the most commonly used drug. By using only hypoglycemic agents, we might have captured patients who had had the disease for a longer duration or who had more severe forms. In addition, not all diseases had similar mortality risk or were equally common. Chronic diseases requiring ongoing care, such as COPD and diabetes, require much more attention by the primary care provider than episodic conditions with less risk of serious complications if left untreated (eg, gout or insomnia). Our study was not designed to estimate the effect of this perceived or true degree of relevance to the provider.

We also only considered data quality in EMRs from one vendor in a single province in a single period of time. Given that there are several other EMR companies operating in Canada, it might also be useful to see if these trends are comparable across the country and among vendors’ products despite there being no evidence to suggest differences between EMR packages in this regard.

While we found no correlation between data quality and “go live” date of the EMR, it could be expected that over time data quality would improve at different rates in different environments. Our study was not designed to describe this change over time.

Interventions and future research

Improving data quality will be an important goal for the future and will require effective inventions. Notably, Greiver et al⁵ have shown that employing a data clerk to synchronize electronic and administrative data was very effective. Longer-term solutions have also been proposed (eg, implementing clinical decision support systems).²³ Developments for chronic diseases such as diabetes²⁴ and asthma²⁵ have recently been reported and have shown potential in better managing health outcomes, as well as improving EMR data quality. More extensive work is needed, either through analytics or algorithmic case detection, as used in the Canadian Primary Care Sentinel Surveillance Network,²⁶ to be able to use primary care EMR problem lists reliably.

Identifying the specific factors that contribute to this variability would therefore be informative for creating strategies to improve problem-list completeness. Quality improvement activities to drive good data will be an ongoing need if primary care clinicians expect to benefit from computable medical information. Given that this study showed relatively low mean data quality, further research into the types of patient, provider, and system characteristics that affect data quality is warranted.

In terms of changing physician behaviour, further research is needed to better understand why some diseases are captured more frequently in problem lists than others.

Conclusion

This study has investigated EMR problem-list completeness related to prescriptions written for chronic diseases in Manitoba. We found that data quality was generally far below expectation for many chronic diseases, except for diabetes in the presence of prescriptions for hypoglycemic drugs. Further, this trend was worse for fee-for-service clinics compared with salaried clinics. Overall, data quality in Manitoba EMR systems will require improvements in order to be used reliably for research or surveillance purposes and to achieve maximum patient care benefits.

EDITOR’S KEY POINTS

• One of the anticipated benefits of the widespread adoption of electronic medical records (EMRs) was their usefulness for quality improvement initiatives and research. However, high-quality data entry and completeness of the data entered in EMRs are required in order to use these data to reliably measure primary care activities and health outcomes.

• This study found that the problem-list field in EMRs in Manitoba did not accurately reflect the types of prescriptions written for patients with chronic disease, and there was considerable condition- and physician-level variability in problem-list completeness. Diabetes was the condition most likely to be documented, and insomnia was least likely. Problem-list completeness for chronic obstructive pulmonary disease and Alzheimer disease or dementia showed the highest variability at the physician level, while insomnia and diabetes showed the lowest physician-level variability.

• Data quality in Manitoba EMR systems will require improvements in order to be used reliably for research or surveillance purposes and to achieve maximum patient care benefits.

POINTS DE REPÈRE DU RÉDACTEUR

• Parmi les avantages anticipés de l’adoption généralisée du dossier médical électronique (DME), on espérait que cela contribuerait à améliorer les soins et qu’on pourrait en utiliser les données pour la recherche. Toutefois, les données consignées dans les DME doivent être complètes et de grande qualité si on veut s’en servir pour évaluer de façon précise le travail effectué en première ligne ainsi que les issues de santé.

• Cette étude a constaté qu’au Manitoba, la liste des problèmes de santé consignés dans les DME ne correspond pas précisément au type de prescriptions que reçoivent les malades chroniques; on notait aussi qu’il y avait une variabilité importante dans l’exhaustivité des problèmes énumérés par les médecins. Le diabète était la maladie la plus susceptible d’être documentée, tandis que l’insomnie était la moins susceptible de l’être. C’est pour la maladie pulmonaire obstructive ainsi que pour la maladie d’Alzheimer ou la démence qu’on observait la plus grande variabilité de la part des médecins, tandis que l’insomnie et le diabète présentaient la moindre variabilité.

• Au Manitoba, la qualité des données consignées dans le système des DME devra être améliorée pour qu’on puisse les utiliser en toute confiance pour la recherche et la surveillance.