How is the electronic health record being used? Use of EHR data to assess physician-level variability in technology use

Abstract

Background

Studies of the effects of electronic health records (EHRs) have had mixed findings, which may be attributable to unmeasured confounders such as individual variability in use of EHR features.

Objective

To capture physician-level variations in use of EHR features, associations with other predictors, and usage intensity over time.

Methods

Retrospective cohort study of primary care providers eligible for meaningful use at a network of federally qualified health centers, using commercial EHR data from January 2010 through June 2013, a period during which the organization was preparing for and in the early stages of meaningful use.

Results

Data were analyzed for 112 physicians and nurse practitioners, consisting of 430 803 encounters with 99 649 patients. EHR usage metrics were developed to capture how providers accessed and added to patient data (eg, problem list updates), used clinical decision support (eg, responses to alerts), communicated (eg, printing after-visit summaries), and used panel management options (eg, viewed panel reports). Provider-level variability was high: for example, the annual average proportion of encounters with problem lists updated ranged from 5% to 60% per provider. Some metrics were associated with provider, patient, or encounter characteristics. For example, problem list updates were more likely for new patients than established ones, and alert acceptance was negatively correlated with alert frequency.

Conclusions

Providers using the same EHR developed personalized patterns of use of EHR features. We conclude that physician-level usage of EHR features may be a valuable additional predictor in research on the effects of EHRs on healthcare quality and costs.

Introduction

The long-term goal of the federal electronic health record (EHR) incentive program is to improve the quality and safety of healthcare.¹ To date, however, research on the effects of EHRs on healthcare delivery has been highly mixed. While many studies have found associations between EHRs and quality improvement, others have found adverse effects, and still others have shown no effect.^2–9 We¹⁰ and others⁸ have argued that the actual effects of EHRs in these studies may have been obscured by a methodological issue, that is, measuring EHR presence or absence as a binary factor. EHRs are complex pieces of technology offering multiple functions and features, and they are embedded in and shaped by complex social environments.¹¹ The same EHR product may be customized differently in different organizations, and implementation processes and organization-specific workflows can also affect how certain features are used. A simple example is that electronic order sets are often developed or modified by healthcare organizations on the basis of local clinical priorities, and thus can vary at the level of the practice or department. Furthermore, it is also highly likely that individual physicians vary in their use of EHR features as a result of preferences or experience. For example, some physicians may habitually use a particular order set as a way of simplifying the ordering process, whereas others may be unaware of it or avoid using it because they disagree with its content or find its design unusable.

For these reasons, effects of EHRs may depend in part on the way EHRs are used by individual clinicians, not merely on whether EHRs are available. One way of assessing individual-level use of EHRs is the meaningful use (MU) ‘objective measures,’ which are intended to capture and promote certain types of EHR use (eg, recording demographics). Another approach is self-report, as has been used in surveys^{8 9 12 13} in which physicians characterize their use of EHR features such as the problem list or radiology result delivery. Lanham and colleagues recently employed interviews and direct observation to distinguish between intensive and less intensive EHR use.¹⁴ However, the availability of EHR data itself creates possibilities for objectively measuring EHR use, capturing granular metrics of usage that could be scaled up or even automated. For example, data capture directly from clinical decision support (CDS) systems has frequently been analyzed to assess rates of response to alerts and reasons for overrides.^15–17 As a number of other commentators have noted, diffusion of innovation (adoption of the technology) differs from infusion (the intensity or sophistication with which the technology is used).¹⁸

Our objective in the current study was to analyze data from an EHR to capture individual physician use of a wider array of EHR functions. We hypothesized that the EHR data would reveal physician-level variation in use of EHR functions, that some of the variation would be linked to provider characteristics, but also that intensity of use of EHR functions would increase over a time period coinciding with the organization's preparation for and early engagement in MU. Finding a high degree of physician-level variability would suggest a high degree of personalization that would support the potential value of measuring physician-level usage patterns in studies of EHR effects. Conversely, lack of variability might suggest that individual usage patterns would not add information to studies of EHR effects.

Methods

Setting

The Institute for Family Health (IFH) is a network of federally qualified health centers (FQHCs) providing safety net care in New York City and less urbanized regions to the north of the city. IFH offers primary care services at 18 sites, with more than 100 physicians (almost all family practitioners) and a patient population of approximately 100 000. IFH has been using the EpicCare EHR since 2003 and is currently a level III patient-centered medical home. EHR customization has taken place at the institutional level and does not vary by clinic, but in some cases clinics may have staffing differences that affect EHR workflow. About half of IFH's family practitioners attested to MU stage I under the Medicaid program in 2012. During the period of the study, two new clinics joined the network, resulting in an increase in numbers of providers and patients.

Metric development

A team of researchers with diverse training (informatics, medicine, biostatistics, and health services research) conducted a literature review and developed a conceptual model broadly categorizing mechanisms through which the EHR might be expected to influence individual provider behavior to affect healthcare quality in the ambulatory setting. The research team then worked with an EHR database analyst at IFH to extract data and construct sample metrics within each of the categories. In this study, we report on 19 sample metrics in five categories (table 2):

1. Use of functions pertaining to the completeness of or access to patient data^{1 8 19–23} (eg, frequency with which providers updated problem lists; time to access test results)

2. Use of decision support^{1 19 24–26} (eg, responses to pushed alerts, reminders, and best practices; also included use of pulled or passive decision support in the form of order sets)

3. Use of electronic ordering (eg, laboratory tests and prescriptions)^{9 27}

4. Use of features for care coordination and communication^{1 19} (eg, secure messaging)

5. Use of panel-level reports^{1 19 25 28} (tools to allow providers to see summarized data about an entire panel).

Table 2.

Eligible providers’ EHR behavior

		Median per provider (25th percentile–75th percentile)					Jan–Jun 2013 only
Metric	All years*	2010	2011	2012	Jan–Jun 2013	p for trend	MU providers	Non-MU providers	p value
No. providers	112	55	70	90	87		41†	46
Data access
% Of encounters per provider with
Problem list marked as reviewed	9.4 (3.8–15.5)	5.1 (0.9–19.9)	7.5 (1.6–13.7)	7.4 (1.9–11.3)	7.5 (4.2–14.6)	0.72	6.0 (3.7–10.0)	10.7 (6.8–20.7)	0.002
Medication list marked as reviewed	5.8 (2.4–15.5)	2.3 (0.1–10.9)	4.0 (0.7–11.1)	4.9 (1.6–10.4)	8.9 (4.5–15.3)	0.05	8.2 (3.6–11.0)	10.4 (5.2–17.1)	0.09
Allergies marked as reviewed	81.0 (73.3–85.1)	78.9 (71.6–82.8)	82.3 (75.2–86.9)	81.9 (73.0–87.6)	81.1 (73.7–86.0)	0.28	81.3 (73.3–86.6)	80.5 (73.7–85.9)	0.93
Problem list altered	23.6 (17.1–30.0)	19.5 (14.7–24.1)	23.0 (17.1–27.7)	22.2 (16.6–29.7)	25.0 (17.3–33.2)	0.002	22.8 (16.9–33.3)	27.7 (20.6–32.3)	0.39
≥1 problem added	23.6 (17.1–30.0)	19.5 (14.7–24.1)	23.0 (17.1–27.7)	22.2 (16.6–29.7)	25.0 (17.3–32.8)	0.002	22.7 (16.9–32.8)	27.6 (20.6–32.3)	0.39
≥1 problem replaced	0.0 (0.0–0.0)	0.0 (0.0–0.0)	0.0 (0.0–0.0)	0.0 (0.0–0.0)	0.0 (0.0–0.0)	0.07	0.0 (0.0–0.0)	0.0 (0.0–0.0)	0.75
≥1 problem dropped	0.0 (0.0–0.0)	0.0 (0.0–0.0)	0.0 (0.0–0.0)	0.0 (0.0–0.0)	0.0 (0.0–0.0)	0.39	0.0 (0.0–0.0)	0.0 (0.0–0.0)	0.36
Days to review laboratory results	4.4 (2.9–7.4)	4.9 (3.0–8.5)	5.3 (3.0–9.8)	4.1 (2.8–7.5)	4.0 (2.5–5.9)	0.01	4.0 (2.4–5.8)	4.0 (2.5–5.9)	0.59
Clinical decision support measures
% Best-practice alerts ‘accepted’	17.7 (9.4–24.6)	18.6 (13.0–27.0)	19.6 (12.6–30.0)	20.9 (10.7–30.0)	18.6 (10.4–26.0)	0.91	22.5 (13.8–28.4)	16.7 (8.6–24.4)	0.03
% Prescribing alerts ‘accepted’	0.1 (0.0–0.2)	0.0 (0.0–0.3)	0.0 (0.0–0.3)	0.0 (0.0–0.2)	0.0 (0.0–0.1)	0.11	0.0 (0.0–0.1)	0.0 (0.0–0.1)	0.91
Drug–allergy alerts	0.0 (0.0–0.0)	0.0 (0.0–0.0)	0.0 (0.0–0.0)	0.0 (0.0–0.0)	0.0 (0.0–0.0)	NT	0.0 (0.0–0.0)	0.0 (0.0–0.0)	NT
Drug–disease alerts	0.0 (0.0–0.0)	0.0 (0.0–0.0)	0.0 (0.0–0.0)	0.0 (0.0–0.0)	0.0 (0.0–0.0)	NT	0.0 (0.0–0.0)	0.0 (0.0–0.0)	NT
Drug–drug interaction alerts	0.1 (0.0–0.3)	0.0 (0.0–0.4)	0.0 (0.0–0.4)	0.0 (0.0–0.2)	0.0 (0.0–0.2)	NT	0.0 (0.0–0.1)	0.0 (0.0–0.2)	NT
Drug–food alerts	0.0 (0.0–0.0)	0.0 (0.0–0.0)	0.0 (0.0–0.0)	0.0 (0.0–0.0)	0.0 (0.0–0.0)	NT	0.0 (0.0–0.0)	0.0 (0.0–0.0)	NT
Ordering
Laboratory tests ordered per encounter	1.0 (0.7–1.4)	0.7 (0.4–0.9)	0.8 (0.5–1.1)	1.2 (0.8–1.5)	1.2 (0.9–1.6)	<0.001	1.2 (0.7–1.6)	1.3 (0.9–1.6)	0.44
Number of order set uses per encounter	1.1 (0.8–1.4)	1.1 (0.9–1.4)	1.1 (0.9–1.6)	1.1 (0.8–1.5)	1.1 (0.8–1.3)	0.61	1.1 (0.8–1.3)	1.0 (0.8–1.3)	0.36
Care coordination measures
% Encounters with after-visit summary printed	59.2 (26.9–83.6)	10.8 (3.2–33.5)	47.8 (21.3–74.2)	73.1 (28.6–90.9)	89.5 (67.1–96.0)	<0.001	94.9 (89.5–97.6)	72.7 (37.0–90.8)	<0.001
Patients messaged/month	1 (0–3)	1 (0–3)	1 (1–4)	1 (0–4)	1 (0–4)	0.53	1 (0–5)	1 (0–3)	0.16
Provider-initiated messages/month	1 (0–4)	2 (0–4)	1 (1–5)	1 (0–4)	1 (0–4)	0.46	1 (0–6)	1 (0–3)	0.20
% Patient-initiated messages responded to	100 (98–100)	100 (100–100)	100 (99–100)	100 (98–100)	100 (99–100)	0.12	100 (94–100)	100 (100–100)	0.35
% Of patient panel using portal during year	6.9 (3.6–13.2)	11.3 (6.4–26.1)	10.8 (5.2–19.9)	8.2 (4.0–14.3)	6.0 (2.8–11.5)	<0.001‡	6.8 (3.8–14.5)	4.9 (2.5–11.3)	0.14
% Of portal users using portal ≥15 times/year	57.6 (51.6–64.7)	55.3 (50.9–65.6)	59.8 (53.2–67.1)	64.4 (54.3–72.1)	54.7 (43.8–66.0)	0.001‡	50.0 (41.8–58.8)	57.2 (48.2–66.7)	0.08
Population/panel level
Viewed panel-level reports (median number of views)	0.0 (0.0–3.8)	NA	0.0 (0.0–5.0)	0.0 (0.0–5.0)	0.0 (0.0–1.0)	0.43	0.0 (0.0–3.0)	0.0 (0.0–0.0)	0.01
MU objective measures§
Core
Core 1—CPOE	100 (99–100)	NA	100 (99–100)	100 (99–100)	100 (99–100)	DNC	100 (100–100)	100 (99–100)	0.42
Core 3—Problem list	97 (95–98)	NA	96 (95–97)	97 (95–98)	98 (96–99)	0.02	98 (97–99)	98 (95–99)	0.51
Core 4—E-prescribing	68 (40–84)	NA	71 (44–84)	75 (46–87)	76 (50–87)	0.26	77 (64–88)	68 (37–86)	0.03
Core 5—Medication list	96 (93–97)	NA	95 (93–97)	95 (94–97)	97 (95–98)	0.01	97 (95–98)	97 (95–98)	0.67
Core 6—Allergy list	99 (99–100)	NA	99 (99–100)	99 (99–100)	99 (99–100)	0.17	99 (99–100)	99 (99–100)	0.44
Core 7—Demographics recorded	94 (91–97)	NA	94 (91–96)	94 (91–96)	95 (91–97)	0.43	96 (93–97)	95 (91–97)	0.94
Core 8—Vitals recorded	97 (96–98)	NA	97 (96–98)	98 (97–98)	98 (97–98)	<0.001	98 (97–98)	98 (97–98)	0.79
Core 9—Smoking status recorded	98 (97–99)	NA	98 (96–99)	99 (98–99)	99 (98–99)	0.75	99 (98–99)	98 (97–99)	0.36
Core 11—Patient copy of health information	100 (100–100)	NA	–¶	–¶	100 (100–100)	NT	100 (100–100)	100 (100–100)	>0.99
Core 12—After-visit summary	63 (33–78)	NA	42 (28–64)	64 (31–81)	72 (49–84)	<0.001	77 (66–85)	65 (38–81)	0.01
Menu
Menu 7—Timely access to data for patients	83 (76–90)	NA	73 (70–79)	81 (77–87)	91 (84–98)	<0.001	85 (83–91)	95 (88–100)	<0.001
Menu 8—Patient education	65 (57–75)	NA	56 (47–66)	69 (60–79)	73 (64–81)	<0.001	77 (69–85)	70 (60–75)	0.001
Menu 9—Medication reconciliation	15 (8–29)	NA	10 (3–19)	13 (5–27)	21 (11–38)	<0.001	18 (9–35)	25 (13–48)	0.18
Menu 10—Summary of care at transitions	77 (36–91)	NA	51 (25–78)	80 (39–94)	85 (58–97)	<0.001	92 (80–97)	70 (34–95)	0.01

For definitions of ‘accepted’ for best-practice alerts and prescribing alerts, please see the Methods section.

*Pooled results across all years are reported, except for panel-level reports and MU measures, for which weighted averages are reported.

†Although 42 providers attested at the end of 2012, one did not contribute data to the 2013 data set and is excluded from this comparison.

‡Comparisons do not include 2013 data as half-year results would not be expected to be similar to full-year results.

§Percentage of patients for whom metric was met, except for e-prescribing metric, which is calculated as a percentage of prescriptions. Core 2, 10, and 13 were accomplished at the organizational level and are not included as they do not vary by provider.

¶No provider had data for this measure in 2011 or 2012.

CPOE, computerized provider order entry; DNC, (model) did not converge; EHR, electronic health record; MU, meaningful use; NA, not available; NT, not tested.

In the EHR system being studied, best-practice alerts were pushed to the provider at entry to the patient record with information about preventive care or other reminders. Prescribing alerts were interruptive and were pushed at order entry. Best-practice and prescribing alerts captured physician responses in different ways. For best-practice alerts, we considered the alert to be accepted if the provider clicked ‘accept,’ or opened the order set highlighted in the alert, or clicked to send a secure message through the EHR. For prescribing alerts, we considered the alert accepted if the provider indicated the drug had been discontinued or the order had been removed. (Alerts that were dismissed or overridden were considered overrides for the purpose of the current study, even if the provider gave a reason for the override.) We also classified best-practice alerts into disease/condition categories (asthma-related alerts, hyperlipidemia-related alerts, etc.). Prescribing alerts were classified by the vendor into types (drug–drug, drug–food, drug–allergy, drug–condition, and drug–alcohol) but not by pharmaceutical class or severity. Drug–alcohol alerts were not included in this study as these alerts were generally intended to be informative rather than to change prescribing behavior. The panel-level reporting function was implemented starting in 2011.

Study sample and data sources

The study sample included all IFH family practitioners who met the criteria for ‘eligible provider’ under the Medicaid MU program and who had at least one documented patient encounter between January 1, 2010 and June 30, 2013, as well as all patients seen by these providers. Each provider was included in the data set for the year during which he/she saw patients and therefore the included set of providers varied slightly year to year. We included both providers who attested to MU stage I in 2012 as well as those who had not done so by the time data collection was concluded. Encounter-level data for the same period were retrieved from the EHR database, including billing diagnoses, problem lists, orders, etc, as well as patient use of the electronic patient portal and messaging. Proprietary logic from the EHR vendor (following the criteria established by the Centers for Medicare & Medicaid Services (CMS)²⁹) was used to create reports of the MU ‘objective measures’ for 2011, 2012, and 2013 for all included providers (the algorithms could not be applied to the 2010 data). In the current study, MU objective measures were reported for both the providers who attested and those who did not. To assess patient illness burden, the Johns Hopkins Aggregated Diagnosis Groups (ADG) count of comorbidities was computed.³⁰ Although the ADG algorithm was designed to be applied to claims data, we instead applied it to the encounter/billing diagnoses for each patient captured in the EHR (without considering any information in the EHR problem list).

The analysis of data from all clinicians allows conclusions about EHR use at the institutional level. To further explore individual-level changes over time, we repeated analyses on the subset of providers who contributed data to all 3.5 years of data collection.

Analysis

All metrics were reported at the provider level on the basis of all patient encounters during each calendar year from 2010 through June 2013 and also pooled across the study period. Categorical variables (such as frequency of updates to the problem list) were calculated as the proportion of encounters per provider. Continuous variables were reported as the mean and SD (for normally distributed variables) or median and IQR (for skewed variables) across encounters for that provider. For table 2, medians were reported for all variables because most were skewed. Descriptive statistics were used to examine the first hypothesis, that EHR data would reveal physician-level variation in use of EHR functions.

For the second set of hypotheses (that some of the variation would be linked to characteristics of the provider), we elected to reduce potential type I errors from multiple hypothesis testing by selecting a limited number of individual hypotheses to test. After discussion among the research team, we formulated three sets of hypotheses that appeared to have high face validity:

1. Response rates to decision-support and best-practice alerts would fall as alert receipt frequency rose, in a dose–response relationship. We hypothesized that providers who saw a particular condition or patient population frequently would be expected to become familiar with the standard of care and rely less upon support offered by the EHR. Conversely, providers who saw a particular condition less often might rely upon EHR guidance more frequently.

2. Alert response rates would be negatively correlated with panel size. Specifically, we hypothesized that information overload or alert fatigue would be likely to be associated with larger panel sizes and would likely reduce response rates.

3. Problem lists would be more likely to be updated for certain types of patients. Specifically, providers with sicker patients on average would be more likely to update problem lists as patients would have more medical problems. Providers with higher proportions of patients using the patient portal would be more likely to update problem lists because patients could use the portal to see their own problem lists. Finally, we hypothesized that providers would be more likely to update problem lists for patients they were seeing for the first time than for established patients as they would be more likely to perceive a need to update the patient record for new patients.

Hypotheses A, B, and C were assessed by generalized linear mixed models (GLMM) using a gamma distribution. Bivariate GLMM were adjusted for within-provider clustering and were weighted by year (ie, 2013 data were weighted at 0.5 because only 6 months of data were included). Multivariable models were additionally adjusted for MU attestation (yes vs no), provider gender, degree (Doctor of Medicine (MD)/Doctor of Osteopathic Medicine (PO) vs Advanced Practice Nurse (APN)), number of patients, and average ADG of the patient panel. Results are reported as incidence rate ratios (IRR) per 100 alerts or per 100 patients, interpreted as the extent to which the likelihood of the outcome changed for every 100 additional alerts or 100 additional patients.

For the third hypothesis, that intensity of use of EHR functions would increase over time, we performed tests for trend using GLMM, adjusted for clustering within physician. Data were modeled using gamma distributions for positively skewed data, beta distributions for negatively skewed data, or normal distributions. As data were available only for half of 2013, models were either weighted by year or else 2013 data were excluded for metrics that represented averages over a 12-month span (as indicated in table 2).

All analyses were conducted in SAS V.9.3 (Cary, North Carolina, USA).

Approvals

The study was approved by the institutional review boards of both Weill Cornell Medical College and the IFH. A waiver of consent was granted as providers and patients were both deidentified in the data sets.

Results

The complete data set contained 112 family practitioners and 430 803 encounters with 99 649 unique patients (table 1). Forty-two providers had attested for MU by the completion of data collection, with the remaining providers planning to attest at future dates.

Table 1.

Provider and patient characteristics

Characteristic	Total or weighted average across study period
Providers, n	112
Attested in 2012, n (%)	42 (37.5)
Encounters per provider, median (25th–75th percentile)	2867 (934–6148)
Unique patients seen per provider, median (25th–75th percentile)	1375 (605–2548)
ADG comorbidity count of panel, median (25th–75th percentile)	0.3 (0.2–0.4)
Female sex, n (%)	71 (63.4)
Credential, n (%)
MD/DO	93 (83.0)
Advanced Practice Nurse	19 (17.0)
Patients, n	99 649
Encounters per patient, median (25th–75th percentile)	2 (1–5)
Mean age (SD)*	36 (20)
Insurance type, n (%)†
Medicaid fee-for-service or managed care	33 339 (36.5)
Medicare fee-for-service or managed care	9635 (9.7)
Commercial	30 134 (30.2)
Uninsured/self-pay	19 394 (19.5)
Other	3905 (3.9)
Female sex, n (%)	58 694 (58.9)
Race, n (%)
White	34 035 (34.2)
Black or African American	24 111 (24.2)
Other	27 695 (27.8)
Unreported/refused	13 808 (13.9)
Ethnicity, n (%)
Hispanic or Latino	30 507 (30.6)
Not Hispanic or Latino	52 997 (53.2)
Not collected/unknown	16 145 (16.2)
Preferred language, n (%)
English	79 369 (79.7)
Spanish	7757 (7.8)
Other language	475 (0.5)
Not collected/unknown/declined	12 048 (12.1)

Percentages may not total 100% due to missing data.

*Mean age represents mean age in 2013.

†Distribution of insurance types represents last known insurance for all patients.

ADG, Aggregated Diagnosis Groups (Johns Hopkins); DO, Doctor of Osteopathic Medicine; MD, Doctor of Medicine.

Metric variability

Median usage varied widely from metric to metric (table 2). For example, prescribing alerts were generally overridden, whereas best-practice alerts were accepted about 20% of the time. Problem lists and medication lists were rarely marked as reviewed (<10% for both), whereas allergies were marked as reviewed more than 75% of the time. Only about two patients were messaged per month, but patient-initiated secure messages were almost always responded to (median response rate per provider: 100%). Providers updated patient problem lists in a median of 20–25% of encounters, virtually always by adding problems rather than dropping them. The panel-level view, which allowed providers to build reports on the basis of their entire patient panel, was used by only a few individuals.

Also, as hypothesized, provider-level variability was also quite high in most of the metrics (table 2). Table 2 shows the IQR to represent the central half of the data, but the full ranges were much wider. For example, best-practice alert acceptance rates per provider ranged from 0% to 68%, the annual average proportion of encounters with the problem list updated ranged from 4.9% to 60.2% per provider, and the annual average proportion of drug–allergy alerts that prompted a drug discontinuation ranged from 0.0% to 62.5% per provider.

Associations with alert firing frequency and characteristics of providers and provider–patient relationship

Overall, as a provider received more best-practice alerts, his or her alert acceptance rate decreased modestly but statistically significantly. The multivariable adjusted IRR was 0.99 per 100 alerts (p<0.001), indicating that for every additional 100 alerts received, the provider accepted 1% fewer of them. Thus, a provider who received 7000 alerts (and about one quarter of providers received 7000 alerts or more annually) was 30% less likely to accept these alerts than a provider who received 4000 alerts, the approximate median for the group (table 3). Similar correlations held within 14 of the 18 categories of alerts evaluated in bivariate GLMM models; 13 of these associations remained significant in models that also adjusted for MU attestation, provider gender, degree, panel size, and average ADG of patient panel (table 3). Drug alerts showed a negative association between frequency and alert acceptance in the bivariate models, but the association was not statistically significant in the multivariable model . Response rates for best-practice alerts were not significantly correlated with panel size (p>0.05), but drug alerts were slightly less likely to result in drug discontinuation as panel size increased (multivariable adjusted IRR 0.94 per 100 patients; p=0.005).

Table 3.

Effect of alert frequency on alert acceptance rate, by alert type

			Change in alert acceptance rate for every 100 additional alerts received
			Bivariate model		Multivariable (adjusted) model
Clinical decision support measure	Median alerts received per provider per year (Q1–Q3)	Number of providers	Incidence rate ratio (95% CI)	p Value	Incidence rate ratio (95% CI)	p Value
Best-practice alert type
Total	4227 (2057–6897)	112	0.997 (0.995 to 0.999)	<0.001	0.991 (0.987 to 0.994)	<0.001
Pediatric immunization	501 (243–951)	112	0.90 (0.88 to 0.92)	<0.001	0.85 (0.82 to 0.88)	<0.001
Diabetes	428 (181–799)	111	1.00 (0.99 to 1.01)	0.74	1.01 (0.99 to 1.03)	0.38
Influenza	491 (219–752)	112	0.97 (0.95 to 0.99)	0.007	0.90 (0.87 to 0.95)	<0.001
Depression	344 (154–575)	112	0.98 (0.95 to 1.01)	0.18	0.98 (0.92 to 1.05)	0.57
Cervical cancer screening	237 (113–389)	109	0.79 (0.74 to 0.84)	<0.001	0.69 (0.63 to 0.76)	<0.001
Adult overweight/obesity	198 (92–408)	112	0.95 (0.86 to 1.05)	0.30	1.09 (0.97 to 1.23)	0.12
Tobacco	171 (71–317)	111	0.73 (0.67 to 0.80)	<0.001	0.59 (0.52 to 0.68)	<0.001
Hyperlipidemia	120 (53–252)	108	0.87 (0.82 to 0.92)	<0.001	0.94 (0.87 to 1.02)	0.16
Colorectal cancer screening	119 (61–236)	108	0.79 (0.73 to 0.82)	<0.001	0.78 (0.72 to 0.84)	<0.001
Pediatric overweight/obesity	71 (16–192)	105	0.78 (0.71 to 0.86)	<0.001	0.64 (0.56 to 0.74)	<0.001
Hypertension	57 (23–110)	108	0.38 (0.26 to 0.55)	<0.001	0.48 (0.29 to 0.80)	0.01
Breast cancer screening	51 (27–91)	108	0.43 (0.36 to 0.50)	<0.001	0.40 (0.33 to 0.49)	<0.001
Pneumonia	40 (16–83)	85	0.58 (0.51 to 0.67)	<0.001	0.58 (0.50 to 0.68)	<0.001
Asthma	24 (6–117)	112	0.68 (0.64 to 0.73)	<0.001	0.64 (0.60 to 0.70)	<0.001
Chlamydia	11 (4–24)	33	0.67 (0.10 to 4.48)	0.23	–	–
Alcoholism	11 (2–48)	23	0.55 (0.44 to 0.68)	<0.001	0.51 (0.37 to 0.70)	<0.001
Anticoagulation	7 (3–16)	92	0.01 (0.004 to 0.04)	<0.001	0.04 (0.01 to 0.13)	<0.001
All others	1091 (591–1879)	112	0.97 (0.97 to 0.98)	<0.001	0.94 (0.92 to 0.96)	<0.001
Drug alert type
Total	1527 (595–2568)	112	0.98 (0.96 to 0.99)	<0.001	0.99 (0.97 to 1.00)	0.12
Drug–drug	931 (399–1724)	112	0.97 (0.95 to 0.99)	<0.001	0.98 (0.96 to 1.00)	0.11
Drug–food	482 (192–797)	111	0.80 (0.74 to 0.87)	<0.001	0.84 (0.70 to 1.01)	0.06
Drug–allergy	25 (10–53)	107	0.16 (0.06 to 0.39)	0.002	0.35 (0.12 to 1.01)	0.05
Drug–disease	4 (2–9)	97	–	–	–	–

Totals are italicized. Bivariate generalized linear mixed model (GLMM) was adjusted for clustering within provider and was weighted by year. Multivariable GLMM additionally included MU attestation in 2012, provider gender, degree (MD/DO vs APN), panel size, average ADG of panel. Cells marked with an ndash represent models that did not converge due to insufficient sample size. The incidence rate ratios (IRR) quantify the increase or decrease in the percentage of alerts accepted for every 100 additional alerts received; an IRR of 0.95 indicates that an alert type was accepted 5% less frequently for every additional 100 times it fired.

ADG, Aggregated Diagnosis Groups; APN, Advanced Practice Nurse; DO, Doctor of Osteopathic Medicine; MD, Doctor of Medicine; MU, meaningful use.

Providers were more likely to update problem lists for a patient they were seeing for the first time than for a patient they had seen before (problem list updated for a median of 26.1% of new patient encounters compared to 20.5% of established patient encounters; p<0.001). Problem list updates were not significantly associated with proportion of patients using their portal account or with average ADG count for the patient panel.

Forty-two providers attested for MU in 2012, but only 41 saw patients at IFH in 2013 and were included in the 2013 data set. As shown in table 2, providers who attested for MU had higher scores than non-MU providers on four of the MU measures and lower scores on one. Almost all the remaining MU measures had ceiling effects, with median compliance rates higher than 95%. The MU providers also had higher usage than non-MU providers for a few of the novel metrics.

Change in usage metrics over time

Tests for trend including data from all providers showed that median usage on seven of the 19 novel metrics changed significantly over the study period (table 2). For example, the proportion of encounters with problem lists updated increased from 19.5% to 25.0% (p=0.002) and the median number of days required to complete a review of a new laboratory test result dropped from 4.9 to 4.0 (p=0.01). The proportion of patients using the portal decreased slightly in 2010–2012 (table 2), probably due to portal enrollment not keeping pace with the increasing patient population as several new centers were added to the FQHC network. However, among portal users, there was a significant increase over time in the proportion who used their account frequently (defined as 15 times per year, which was the top quartile of use).

In addition, median performance on eight of the 14 MU objective measures also rose significantly. (Remaining measures that did not rise significantly either had ceiling effects or missing data. In the case of e-prescribing, the rate already exceeded the minimum established by MU at baseline.)

As the set of providers included in each year's data was somewhat different, these findings should be interpreted as institution-level trends not provider-level trends. Providers contributed a mean of 2.7 years (SD 1.3) of data to the data set. However, 46 providers contributed data to all 3.5 years of the study and were included in a longitudinal subset analysis (see online supplementary appendix). Trends of increasing EHR usage within this longitudinal subset were almost identical to trends for the institutional-level analysis with all providers, in that all the same trends were statistically significant with effect sizes of similar magnitude with the following exceptions: (1) days to review laboratory test results continued to show a reduction over time, but the effect was no longer statistically significant (p=0.08); and (2) the rate of problem list review, which did not change significantly over time for the larger group, significantly declined over time in the subset (p=0.02). In the subset analysis, two of the metrics could not be modeled due to insufficient sample size or lack of convergence of the model (proportion of patients using portal ≥15 times per year, and the MU medication list metric).

Discussion

Primary care providers using the same EHR system in a network of clinics developed personalized approaches to their use of EHR features, which varied along dimensions ranging from frequency with which they updated patient problem lists to responsiveness to CDS alerts. Some of the variability was associated with provider or patient–provider characteristics; for example, providers behaved differently with new patients than with established patients, and responded differently to uncommon alert types than to common ones. A certain amount of individual-level variability was evident in the MU objective metrics, even after 42 IFH providers attested for MU in 2012, but the novel metrics developed for the current study showed even greater variability, presumably because there were no incentives associated with them. Some of the EHR functionalities measured here showed modest changes over the 3.5 years of this study. While some were directly linked to MU incentives (eg, the increase in use of after-visit summaries), in other cases there was no direct link (such as the drop in the number of days required to review laboratory test results).

One implication of these findings is that because individual-level use of EHR features to accomplish tasks varies, capturing this variation may add value to research on the quality and cost effects of health information technology.¹⁰ One approach to examining individual-level use of EHRs is to employ the MU ‘objective measures.’ The stage I objective measures may, however, be too basic to use in a nuanced examination of provider behavior. For example, providers can achieve the MU problem-list metric by ensuring that patients have a single problem listed, but do not need to show that the problem list is complete or up-to-date. Similarly, providers can achieve the MU decision-support metric by demonstrating that the decision support functionality is enabled, but do not need to show that they have accepted any of the alerts.

Another approach to capturing and evaluating individual-level use of EHRs is by administering surveys^{8 9 12} asking users to characterize the availability and use of EHR features. This approach allows a more granular analysis of the components of the EHR that might influence care. Poon et al employed a survey approach to demonstrate that EHR use as a binary factor is not associated with ambulatory physician quality performance, whereas use of specific EHR features was associated with significantly better quality scores.⁸ Somewhat similarly, Amarasingham and colleagues used a survey tool to identify specific elements of the EHR associated with hospital quality and costs.⁹ A very rich approach to assessing EHR use patterns was developed by Lanham and colleagues, who used interviews and direct observations to identify patterns of high, medium, and low EHR usage.¹⁴

Our approach is novel in that we employed EHR data to measure EHR use at the provider level, an approach that is potentially scalable. In addition, we attempted to develop sample metrics that might be expected to be associated with quality effects. For example, rather than recording only whether the patient problem list was populated, we captured whether the list was updated from encounter to encounter. Data capture directly from health IT systems has previously been explored for domain-specific studies, especially in CDS.^{15–17 22 31 32} However, our goal was to develop sample metrics in a variety of different domains, so that alert override rates could be placed in the context of the larger issue of use of the EHR.

It is important to note that the metrics developed here should be interpreted as measures of intensity of use of the EHR, not necessarily as measures of better use of the EHR. For example, the electronic laboratory ordering metric (which increased significantly over time) should be seen as a measure of intensity of EHR use, not as a quality indicator. The high alert override rate also cannot necessarily be interpreted as an indicator of good or bad use of the EHR as it is consistent with the high override rates reported in an array of studies on contemporary CDS systems; ^{15–17 22 31 32} alert overrides can be attributed to factors ranging from alert fatigue to lack of clinical relevance in a particular case, and substantial proportions are found to be clinically appropriate on review.³³ The alert override rate is also dependent upon the severity threshold of the alerts shown to providers, which is generally determined at the institution level; unfortunately, we did not have severity or drug-class data on the alerts. Use of the patient portal is another metric that has not been definitively linked to improved quality.³⁴

The provider-level variability observed here is most likely caused by a wide variety of factors at the product, organizational, provider, and patient levels. At the product level, system users reported that the panel-level view function in the EHR was rarely employed because it was difficult to use and training on this feature was not offered throughout the network. At the organizational level, allergies were most likely marked as reviewed more frequently than other elements of the record because the allergy review was typically performed by a medical assistant during patient intake, whereas other components (such as medications and problem lists) were reviewed by the provider during the encounter. Furthermore, during the study timeframe, new clinics joined the network and underwent EHR rollouts, resulting in subsets of providers who had lower familiarity with the EHR, as well as patients being newly introduced to the electronic patient portal. Although the EHR product was standardized across clinics, site-to-site staffing differences (not captured in our data) may have influenced EHR workflow. Providers who attested for MU, as expected, had higher scores on most MU measures that were not already affected by ceiling effects. However, they differed from the non-MU providers on only a few of the non-MU measures, suggesting the possibility that they were responding to specific incentives rather than demonstrating overall higher intensity of EHR use.

At the provider level, the data showed negative correlations between alert frequency and alert acceptance, both overall and by type. This finding is compatible with the explanation that providers are more likely to accept recommendations about conditions that they see less frequently and therefore are less familiar with. However, the findings are also compatible with an explanation of alert fatigue.^{32 35–37}

Limitations

The metrics here were developed de novo for the current study and have not been validated with other data sets or demonstrated to be associated with healthcare quality. Limited information was available on the providers themselves in order to maintain confidentiality. The observed variability in usage may have been linked with other variables that we were unable to capture for the current analysis, including provider age and the provider's practice site at each time point in the study. A number of patient-level characteristics are likely associated with these metrics, but patient-level analyses are outside the scope of the current study. This study was conducted at a single healthcare organization using a particular commercial product and may not be generalizable to other types of organizations or products.

Conclusions and implications

EHRs, like other information technologies, cannot be expected to cause the intended effects on healthcare delivery unless their features are regularly used.^{10 38} Primary care providers at a single center using the same EHR product varied in their use of available functions of the EHR. This suggests that individual-level measures of usage may add value to future research on quality and cost outcomes of EHR use, even though more work remains to be done on explanations of this variability.