Implementation of electronic medical records

Abstract

Objective

To study the effect of electronic medical record (EMR) implementation on preventive services covered by Ontario’s pay-for-performance program.

Design

Prospective double-cohort study.

Participants

Twenty-seven community-based family physicians.

Setting

Toronto, Ont.

Intervention

Eighteen physicians implemented EMRs, while 9 physicians continued to use paper records.

Main outcome measure

Provision of 4 preventive services affected by pay-for-performance incentives (Papanicolaou tests, screening mammograms, fecal occult blood testing, and influenza vaccinations) in the first 2 years of EMR implementation.

Results

After adjustment, combined preventive services for the EMR group increased by 0.7%, a smaller increase than that seen in the non-EMR group (P = .55, 95% confidence interval −2.8 to 3.9).

Conclusion

When compared with paper records, EMR implementation had no significant effect on the provision of the 4 preventive services studied.

Electronic medical records (EMRs) have been identified as critical to quality improvement efforts,^1–4 and policies favouring the establishment of EMRs are being implemented in the United States and in most Canadian provinces.^2,5,6 In 2005, the Ontario government offered a subsidy to selected primary care physicians for the purchase of EMRs.⁵ Physicians who were eligible were those who had chosen a blended capitation payment system based largely on the age and sex of patients enrolled in their practices. By the end of the program in 2009, most physicians offered the subsidy had purchased EMRs.⁷ This provided an opportunity to compare groups of physicians in the same communities implementing and not implementing EMRs.

Although EMRs invite hope for improvement,⁸ there is still much that is unknown about the effect of EMRs on quality and performance. A recent systematic review found that computerized decision-support systems improved practitioner performance, especially with regard to immunizations.⁹ However, these systems were not commercial, off-the-shelf EMRs like those commonly found in primary care practices. Several studies have found that EMRs might not improve care^10–15 and might even be a source of errors.^15–17 Studies of EMR implementation have often been descriptive, with very few evaluating measurable outcomes.¹⁶ At present, the effect of EMR implementation on the quality of care in small community-based family practices is unknown.

Electronic medical record systems are complex, and their implementation involves changes to many processes; only some changes can be implemented early on. In this study, we focused on preventive services targeted by Ontario’s pay-for-performance program as markers of early progress in EMR implementation. These services were attached to financial incentives and were perceived as representing good care.¹⁷ The pay-for-performance target numbers varied by physician and were based on the percentage of enrolled family practice patients being provided with Papanicolaou smears, mammograms, influenza vaccinations, fecal occult blood screening, and primary vaccinations (in children younger than 2 years).^18,19

In order to be eligible for funding in Ontario, an EMR system had to include the ability to manage the provision of such preventive services through point-of-care alerts and reminder letters. Although physicians using paper records were also offered financial incentives and were expected to produce an increase in preventive services, those using EMRs had electronic tools that could lead to a greater increase.

Our objective was to compare the differences in provision of these preventive services between community-based family physicians implementing EMRs and those continuing to use paper-based records.

METHODS

Participants

We followed 2 cohorts of physicians: a group of 18 physicians implementing EMRs and a group of 9 physicians using paper records (the non-EMR cohort). Physicians in both cohorts were community-based, were affiliated with a local hospital, and were located in the Toronto area. All were members of the same local after-hours clinic. Physicians were signed out to the clinic and took turns providing walk-in care at a single location on evenings and weekends. No preventive services were offered at the clinic and no clinic data were included in the study.

The physicians in the EMR cohort had previously participated in a pay-for-performance study,²⁰ so data on their characteristics were already available. They switched to the blended capitation model at the end of 2004 and began EMR implementation in the first 4 months of 2006 using the same EMR software. We studied the change in preventive services in the first 2 years of EMR implementation (2006 and 2007). The principal investigator was also a participant in this study (M.G.).

We recruited the non-EMR physicians through a letter of invitation and brief survey sent to all members of the after-hours clinic who were not part of the EMR cohort (125 physicians). These physicians were not eligible for the EMR subsidy. The recruitment process is shown in Figure 1.

Figure 1.

Recruitment process for the non-EMR cohort

EMR—electronic medical record.

Data sources

The Ontario Ministry of Health and Long-Term Care provides family physicians with a list of patients in their practices who are eligible for each of the 4 preventive care services. We selected charts from these lists, using a random numbers table, from which we recorded the following: each patient’s age and sex, the presence or absence of a service within the required time period, and any record that a reminder letter had been sent to patients overdue for a service. We entered the data into an Epi Info database.²¹

We examined information on factors possibly associated with the provision of preventive services.^22–32 Data were obtained through a questionnaire administered to all physicians. Following the methods of Glazier et al,³³ practice-level data were also derived from linked administrative databases at the Institute for Clinical Evaluative Sciences (ICES). Income quintiles were derived from 2001 census data linked to patient postal codes. Neighbourhood income quintiles and recent immigration status were derived using published methods.^33,34 Morbidity and comorbidity (adjusted diagnosis groups and resource utilization bands) were derived using the Johns Hopkins Adjusted Clinical Groups software,³⁵ available from ICES, which has been described elsewhere.^36–43 Numbers and percentages of patients with chronic diseases were obtained from linked ICES health databases.^38–44

We audited paper charts for physicians in the non-EMR cohort and obtained data from EMR charts for physicians in the EMR cohort. When data were unavailable in the EMR, we retrieved and obtained data from the paper chart.

Five data auditors abstracted data from both paper charts and EMRs. The research coordinator initially audited 10 charts for each service in 2 practices and reviewed this with the principal investigator (M.G.). The coordinator then trained each data auditor, and reviewed at least 10 charts for each service from each auditor. The data were collected on paper forms and entered in the Epi Info database by 2 data entry clerks. Each clerk entered data from a training sample of at least 10 charts for each service. A randomly selected 10% data sample for each service, each year, and each physician was re-audited and entered in the database; we used the κ statistic to compare the 2 audits.

Outcome measures

The study’s end point was whether or not a preventive service was provided and documented within a required time period for each eligible patient. The target patient population consisted of all eligible patients enrolled in the participating practices. Documentation that the patient received the service through another health care provider was acceptable. Information on services is presented in Table 1.

Table 1.

Eligibility criteria, exclusion criteria, and required period for preventive service provision in a pay-for-performance family practice setting

SERVICE	ELIGIBLE POPULATION	EXCLUSION CRITERIA	REQUIRED PERIOD FOR SERVICE PROVISION*
Papanicolaou smear	Enrolled women aged 35 to 69 y	Previous hysterectomy	Documented service within the past 30 mo before March 31
Screening mammograms	Enrolled women aged 50 to 69 y	History of breast cancer	Documented service within the past 30 mo before March 31
Influenza vaccination	Enrolled patients aged 65 y or older	None	Documented service from October 1st to December 31st of previous y
Fecal occult blood test	Enrolled patients aged 50 to 74 y	History of colorectal cancer; history of inflammatory bowel disease; colonoscopy within the past 5 y	Documented service in the past 30 mo before March 31
5 completed primary immunizations (4 diphtheria, tetanus, pertussis, polio, and Haemophilus influenzae type b vaccinations and 1 MMR vaccination)†	Enrolled children age 30 to 42 mo as of March 31	None	Documented completion of vaccinations within the past 12 mo before March 31

MMR—measles, mumps, and rubella.

^*The fiscal year end in Ontario’s health care system is March 31; therefore, March 31, 2007, would be considered the 2006 year end and March 31, 2008, would be the 2007 year end.

^†Children’s vaccination scores were ultimately excluded from this study as lists of eligible children for 2006 could not be obtained in the non-EMR cohort.

We calculated a composite process score⁴⁴ by using the total number of charts audited for eligible patients for each service per physician as the denominator and the total number of services documented in the audits as the numerator. We could not obtain lists of eligible children for 2006 in the non-EMR cohort. As a result, children’s vaccinations were excluded from the composite score.

We compared the change in the proportion of patients who received preventive services between the EMR and non-EMR cohorts during EMR implementation. A 5% increase was considered the minimum clinically important difference.

Sample size calculation

We calculated the sample size required to have an 80% power to detect a clinically important increase in service provision of 5% or higher²⁰ in the first 2 years of EMR implementation (assuming relatively stable rates in the non-EMR cohort and with rates of influenza vaccination ranging from 83% to 88%) using an α level of .05, and determined that 40 charts per service per provider would be required. To further increase statistical power, we audited 50 charts per year per service per physician. Physicians who practise in groups might influence one another; this would reduce the variation in practice and might inflate the observed effect.⁴⁵ Based on our previous pay-for-performance study,²⁰ the estimated intraclass correlation coefficient⁴⁶ due to clustering of physicians within practices was 0.01. If we assume that, on average, the recruited physicians were clustered within groups of 4 physicians, and that the intraclass correlation coefficient was 0.01, the inflation factor would be 1.03 ([1 + (4 - 1) × 0.01]), which should have a negligible effect on the sample size required.

Analysis

We compared unadjusted percentages using χ² tests and compared adjusted percentages using multivariate logistic regression analysis. We used generalized estimating equations to adjust for the clustering structure of the data in regression models. When comparing the 2 groups, we adjusted for patient age for each individual patient,⁴⁷ physician sex,²⁰ years since each physician’s medical school graduation,²² and Certification in Family Medicine (ie, CCFP) status²⁶ using multivariate logistic regression analysis. Administrative data could not be used for adjustment, as these data were not available to us at the patient level. Percentage of fecal occult blood tests provided might have been different from the other services provided; the provincial provision rate was low and there was a concurrent public health campaign in 2007.^48,49 As a result, we re-analyzed the 2 cohorts with this screen excluded.

As the principal investigator was also a participant in the study, a sensitivity analysis was performed by excluding her practice data.

We performed analyses with SAS software, version 9.1. All tests were 2-sided using an α level of .05. The study was approved by the University of Toronto’s Research Ethics Board; the Sunnybrook Research Institute’s Research Ethics Board approved the use of ICES data. All participating physicians provided written informed consent.

RESULTS

Characteristics of the EMR and non-EMR physicians and their practice populations are presented in Table 2 and Table 3,^{22, 34–43} respectively. Physician characteristics between the 2 cohorts were similar. Patients in the EMR cohort were slightly younger and had lower levels of morbidity and comorbidity.

Table 2.

Characteristics of physicians in the EMR and non-EMR cohorts

VARIABLE	NON-EMR COHORT (N = 9)	EMR COHORT (N = 18)
Year of graduation,* median (range)	1984 (1966 to 1993)	1977 (1964 to 1992)
Men,* n (%)	6 (66)	10 (56)
CCFP,* n (%)	7 (77)	11 (61)
No. of medical doctors per practice,* median (range)	4 (1 to 6)	3 (1 to 6)
No. of hours worked per week,* median (range)	44 (28 to 80)	42 (30 to 60)
No. of patients per physician,* median (range)	1200 (850 to 1600)	1206 (630 to 2200)
No. of Canadian graduates†	8	16

CCFP—Certification in Family Medicine, EMR—electronic medical record.

^*Obtained from self-reports.

^†Obtained from administrative databases.

Table 3.

Comparison of patient characteristics between the EMR and non-EMR cohort practice populations as of August 31, 2007

PATIENT VARIABLE*	NON-EMR COHORT (N = 9)	EMR COHORT (N = 18)
Patients, N (mean per physician)	10 591 (1177)	23 514 (1306)
Age, median (IQR)	47 (31 to 63)	45 (27 to 60)
Men, n (%)	4 767 (45.0)	10 106 (43.0)
Neighbourhood income quintile,22 n (%)
• Unknown	31 (0.3)	51 (0.2)
• 1 (lowest)	1594 (15.1)	3084 (13.1)
• 2	1438 (13.6)	3643 (15.5)
• 3	1951 (18.4)	4345 (18.5)
• 4	2414 (22.8)	5 091 (21.7)
• 5 (highest)	3163 (29.9)	7300 (31.0)
Recent immigrant,22 n (%)	1148 (10.8)	1398 (5.9)
Comprehensiveness of care,22,34 mean (SD)†	0.50 (0.34)	0.54 (0.35)
Overall morbidity (resource utilization bands),35,36 mean (SD)‡	2.81 (1.14)	2.73 (1.02)
• 0 (lowest), n (%)	657 (6.2)	1 047 (4.5)
• 1, n (%)	616 (5.8)	1 480 (6.3)
• 2, n (%)	1 720 (16.2)	4778 (20.3)
• 3, n (%)	5 344 (50.5)	12 567 (53.4)
• 4, n (%)	1 614 (15.2)	2783 (11.8)
• 5 (highest), n (%)	640 (6.0)	859 (3.7)
Overall comorbidity (adjusted diagnosis groups),35,36 mean (SD)§	5.43 (3.48)	4.77 (3.04)
• 0 (lowest level of comorbidity), n (%)	657 (6.2)	1046 (4.4)
• 1–4, n (%)	3962 (37.4)	11 189 (47.6)
• 5–9, n (%)	4615 (43.6)	9502 (40.4)
• ≥ 10 (highest level of comorbidity), n (%)	1357 (12.8)	1777 (7.6)
Common comorbidities, n (%)
• Diabetes37	1041 (9.8)	1934 (8.2)
• CHF40	300 (2.8)	386 (1.6)
• Hypertension38	2823 (26.7)	5594 (23.8)
• MI39	193 (1.8)	311 (1.3)
• Asthma41	1500 (14.2)	3143 (13.4)
• COPD42	626 (5.9)	1120 (4.8)
• Mental health issues43	2391 (22.6)	4937 (21.0)

CHF—congestive heart failure, COPD—chronic obstructive pulmonary disease, EMR—electronic medical record, IQR—interquartile range, MI—myocardial infarction, SD—standard deviation.

^*Obtained from administrative databases.

^†Comprehensiveness of care was determined by measuring the percentage of bills for 21 common services that were provided by the patients’ own family physicians.

^‡Resource utilization bands indicate morbidity and expected health care system use, from 0 (lowest) to 5 (highest).

^§Adjusted diagnosis groups indicate comorbidity, from 0 groups (lowest level of comorbidity) to 10 or more groups (highest level).

Results for individual services are presented in Table 4. Nine hundred charts were audited for each service for each year in the EMR cohort and 450 charts were audited in the non-EMR cohort. The number of Pap smears and mammograms increased more in the EMR cohort, while the rate of influenza vaccinations and fecal occult blood testing increased more in the non-EMR cohort. Composite process scores are shown in Table 5.

Table 4.

Service provision for individual services in the EMR and non-EMR cohorts before and after EMR implementation

SERVICE	COHORT	2006, %	2007, %	DIFFERENCE, %	DIFFERENCE IN CHANGE BETWEEN EMR AND NON-EMR GROUPS
Influenza vaccine	EMR	70.7	69.8	−0.9	−8.3% in EMR cohort
	Non-EMR	59.2	66.5	7.4
Papanicolaou smear	EMR	76.1	79.7	3.6	+1.1% in EMR cohort
	Non-EMR	75.6	78.1	2.5
Fecal occult blood test	EMR	28.7	32.1	3.4	−1.4% in EMR cohort
	Non-EMR	41.1	45.9	4.8
Mammogram	EMR	75.2	80.9	5.7	+6.3% in EMR cohort
	Non-EMR	78.3	77.7	−0.6

EMR—electronic medical record.

Table 5.

Changes in composite process scores for the provision of preventive services between EMR and non-EMR cohorts

PROVISION OF PREVENTIVE SERVICES*	EMR COHORT	NON-EMR COHORT	DIFFERENCE	ADJUSTED DIFFERENCE BETWEEN THE GROUPS†
Including fecal occult blood testing
• Composite process score in 2006, %	63.0	63.6	0.6	NA
• Composite process score in 2007, %	65.7	67.1	1.4	NA
• Difference (95% CI)	2.7 (0.6 to 5.0)	3.5 (0.5 to 6.6)	0.8 (−3.0 to 4.6)	0.7 (−2.8 to 3.9)‡
Excluding fecal occult blood testing
• Composite process score in 2006, %	74.0	71.0	3.0	NA
• Composite process score in 2007, %	76.8	74.1	2.7	NA
• Difference (95% CI)	2.8 (0.5 to 5.1)	3.1 (−0.2 to −6.4)	0.3 (−3.7 to 4.4)	0.3 (−3.0 to 3.6)§

CCFP—Certification in Family Medicine, CI—confidence interval, EMR—electronic medical record, NA—not applicable.

^*Preventive services also include Papanicolaou smear tests, influenza vaccination, and screening mammograms.

^†Adjusted for physician sex, CCFP status, number of years since graduation from medical school, and patient age.

^‡P = .55.

^§P = .53.

Female physician sex and younger patient age were positively correlated with the likelihood of receiving a service. There was no correlation with years since medical school graduation or Certification status. There was no significant difference in the change in service provision between the 2 groups; differences and confidence intervals were less than 5%. The results were not affected by the exclusion of fecal occult blood screening data.

Physicians in the EMR cohort mailed letters to 23 patients overdue for services in 2005, 265 patients in 2006, and 677 patients in 2007. Physicians in the non-EMR cohort mailed 1 letter in 2006 and none in 2007.

The results were unaffected by the exclusion of the principal investigator’s data. The κ statistic for the 10% of charts that were re-audited was 0.954, consistent with an acceptable level of agreement.

DISCUSSION

We found no statistically significant or clinically important difference in the change in preventive service provision between physicians implementing EMRs and those continuing to use paper records. Other studies have found no difference in the change in care of diabetes between EMR- and paper-based practices¹¹ and no improvement when additional experience with EMR accrues over time.^11,13,15

The EMR system we studied included elements that have been associated with an increase in the provision of preventive services, such as point-of-care alerts^9,50,51 and the ability to generate reminder letters to patients.⁵² However, studies have shown that alerts or patient reminders for overdue services, even if available, might not be consistently implemented in practice.^11,15,53,54 We found evidence of reminder letter mailings in the EMR cohort, but this did not appear to have had an effect on service uptake.

To frame and explain our results, a qualitative study presented in an accompanying article⁵⁵ explores the context of this EMR implementation. Similar studies could help to determine what aspects of EMR implementation are associated with improved quality of care.

Limitations

This study was limited to a group of selected physicians in Toronto. However, all physicians in this study were practising in community-based settings, similar to most family physician settings in Ontario.⁵⁶ Physicians in our cohorts were similar to their colleagues in capitated and reformed fee–for-service models in Ontario urban centres.³³

Nineteen percent of physicians using paper records responded to the study invitation; we do not know if their characteristics differed from those of the nonresponders. The participating physicians provided preventive services to a large proportion of their patients. Increases might have been limited by ceiling effects. Also, we studied a single EMR system; results might differ for physicians using other EMR systems.

This was an observational cohort study, and is therefore subject to both measured and unmeasured confounders. In this study, we measured confounders that have been reported in the literature to affect preventive services^{22–32,47,57,58} and used statistical adjustments to adjust for these factors. There were differences between our 2 cohorts in terms of physician funding mechanisms. However, a recent study using administrative data did not find any consistent difference in the provision of preventive services between Ontario physicians in capitated or reformed fee-for-service groups.⁵⁹

Conclusion

Among the family physician practices studied, the first 2 years of EMR implementation were not associated with an increase in the provision of preventive services targeted by Ontario’s pay-for-performance program when compared with the continued use of paper records. It should not be assumed that EMR implementation improves care.

EDITOR’S KEY POINTS

• One of the drivers of electronic medical record (EMR) implementation is the hope that EMRs can improve quality of care for patients.

• The authors examined preventive services with pay-for-performance incentives that should have improved with EMR use owing to electronic tools that assist in timely service provision (eg, alerts, automated reminders).

• There was no difference in service provision between physicians using EMRs and those continuing to use paper records (in the first 2 years of EMR implementation).

• It should not be assumed that EMR implementation improves care.

POINTS DE REPÈRE DU RÉDACTEUR

• Une des raisons qui incitent à adopter les dossiers médicaux électroniques (DMÉ) est l’espoir qu’ils peuvent améliorer la qualité des soins.

• Les auteurs ont examiné les services préventifs qui, avec une rémunération au rendement comme mesure incitative, auraient dû s’améliorer avec les DMÉ grâce aux outils électroniques qui facilitent la prestation de ces services en temps opportun (p. ex. signaux d’alerte, rappels automatisés).

• Il n’y avait pas de différence de prestation des services entre les médecins utilisant les DMÉ et ceux qui continuaient d’utiliser les dossiers papiers (durant les 2 ans suivant l’introduction des DMÉ).

• On ne devrait pas présumer que la mise en œuvre des DMÉ améliore les soins.