Small Physician Practices In New York Needed Sustained Help To Realize Gains In Quality From Use Of Electronic Health Records

Abstract

While Regional Extension Centers and other national policy efforts to increase the adoption of electronic health records (EHR) have been implemented in the United States, the relationship between EHR adoption and quality of care remains poorly understood. We evaluated the early effects on quality of the Primary Care Information Project, which provides subsidized EHRs and technical assistance to primary care practices in underserved neighborhoods in New York City. We find that nine or more months of participation in the Primary Care Information Project is associated with improved quality, but only for certain quality measures and only for physicians receiving extensive technical assistance.

Introduction

The rate of adoption of electronic health records (EHR) for outpatient care in the United States has been slow, but is rapidly accelerating. Between 2009 and 2011, the proportion of outpatient physicians using of a “basic” EHR – an EHR with the ability to generate problem lists, document medications, and view test results - increased from 22% to 35%.(1) Nonetheless, small practices—which constitute the majority of practices in the U.S.(2)— and practices that are owned by physicians have the lowest rates of EHR adoption. (1, 2) Further, some evidence suggests that practices with higher proportions of minority, Medicaid, and uninsured patients are less likely to use EHRs.(3)

In an effort to encourage EHR adoption, the American Recovery and Reinvestment Act of 2009 allocated $643 million for the creation of a Health Information Technology Extension Program. (4) This program provides ongoing technical assistance to practices through Regional Extension Centers, with a focus on small primary care practices. Through a combination of clinical decision support, tool to reduce medical errors (such as eprescribing and clear and comprehensive display of test results), population management through registry generation, and improved communication with patients and other providers, EHRs have tremendous potential to improve quality of care. (5, 6)

Despite the enthusiasm for EHRs, and the recent acceleration in adoption, evidence that EHRs improve quality of care is mixed.(5-13) EHRs may have long term benefits, but the transition from paper to electronic medical records can be disruptive, with a steep learning curve required to effectively use the features of the EHR to improve quality of care. While a recent review identified that 92 percent of recent articles on health information technology – including electronic health records and accompanying technologies, such as clinical decision support – found positive results from implementation,(13) other reviews have emphasized the lack of formal evaluation of health IT applications.(5, 6) Very little is known about the effects of the recently created Regional Extension Centers on quality.(14) Although smaller-scale versions of the Regional Extension Center model have been implemented, evaluations have focused on pre- and post- intervention improvement and have not convincingly demonstrated the effectiveness of this approach.(15-20)

This study uses a unique dataset based on multi-payer medical claims in New York State to evaluate whether the Primary Care Information Project – in its first years as a major electronic health record implementation and technical assistance program – improved quality of care for small practices located in underserved areas in New York City.

The Primary Care Information Project

In 2005, the New York City Department of Health and Mental Hygiene created the Primary Care Information Project (PCIP), which began assisting practices with EHR adoption in late 2007. PCIP subsidized two years of EHR software costs for eligible primary care providers (serving a minimum of 10% Medicaid or uninsured patients) in New York City. These providers were required to adopt the eClinicalWorks™ EHR, which included several functionalities co-developed by PCIP to improve prevention-oriented services. These functionalities include a clinical decision support system, an enhanced patient registry that allows the providers to search on specific patient characteristics, and e-prescribing. PCIP also provided technical assistance and coaching on quality improvement with a focus on preventative care, based on the NYC health policy agenda “Take Care New York”.(21-23) PCIP staff provided practices with on-site technical assistance visits lasting after the implementation of EHR software. Technical assistance focused initially on helping practices to address barriers to effective EHR use and to troubleshoot EHR implementation issues. Subsequent technical assistance visits assisted practices to use the EHR to improve the population health of their panel, with an emphasis on adult patients with cardiovascular health problems. This included interfacing with the patient registry, using patient flow sheets, and generating patient order sets, custom alerts, and quality reports.

Since July of 2007, PCIP has enrolled over 3,300 physicians in over 600 practices, making it the largest community-based EHR implementation and extension program in the U.S.(21) In 2011, PCIP became one of the 60 Regional Extension Centers established nationally.(24)

Methods

We tested whether physicians in PCIP improved outpatient quality more than a set of matched comparison physicians in New York State that did not participate in PCIP. We also examined whether the effect of PCIP was greater for physicians who received more technical assistance from the program. This study encompasses the time period before, and up to two years after EHR implementation for PCIP physicians.

Data

To generate measures of the quality of care, we used data from the New York Quality Alliance dataset for calendar years 2007-2010. The dataset includes outpatient quality of care measures based on Healthcare Effectiveness Data Information Set specifications from 13 private insurers (including some managed Medicare and Medicaid plans) for physicians in New York City, Long Island, the Hudson Valley, the Capital Region (surrounding Albany), and Western New York State (Exhibit 1).(25) Together, participating plans make up approximately 40% of the privately insured market in New York State.(26) In addition to containing numerators and denominators for each quality measure for 6,756 physicians with at least 30 eligible patients for a given measure in a given year, the dataset also contains information on physicians’ location of practice, and specialty.

Exhibit 1.

Characteristics of physicians participating and not participating in PCIP for all eligible physicians and the propensity score matched sample

	PCIP physicians	All Comparison physicians	Matched Comparison physicians
N	360	4,645	360
Location*Ψ (%)
New York City	100	43.7	90.8
Long Island	0	23.3	5.0
Hudson Valley	0	17.8	1.9
Capital Region and Northern New York	0	12.4	0.6
Western New York	0	2.8	1.7
Percent poverty in zip code†*(mean, sd)	20.1 (6.7)	13.6 (8.0)	20.0 (6.9)
Number of physicians in practice* (mean, sd)	2.1 (2.7)	5.4 (16.8)	1.8 (1.9)
Specialty* (%)
Internal medicine	41.4	41.2	41.7
Family practice	15.8	21.4	14.7
Pediatric medicine	19.2	16.4	21.7
Cardiology	2.8	3.1	2.8
Pulmonology	3.6	1.8	2.2
Gastroenterology	3.1	2.0	3.9
Other	14.2	14.2	13.1
Degree* (% with MD)	94.4	91.0	92.5
Practice ownership* (% with corporate parent)	6.7	26.0	4.4
Number of technical assistance visits¥ (mean, sd)	3.1 (3.7)	-	-
Exposure to PCIP¥ (%)
Less than 6 months	37.8	-	-
6 to less than 12 months	12.5	-	-
12 to less than 18 months	7.8	-	-
18 to less than 24 months	15.3	-	-
24 months or more	26.7	-	-

SOURCE: Authors’ analysis

^†Note 1: For zip code in which practice is located

^¥Note 2: As of December 31, 2010

^*Note 3: p<.05 for test of difference between PCIP physicians and all comparison physicians

^ΨNote 3: p<.05 for test of difference between PCIP physicians and matched comparison physicians

We used data from the New York City Department of Public Health and Mental Hygiene on PCIP physician go-live dates, technical assistance received after EHR implementation (from 2008 onwards), physicians’ location of practice, and type of practice (small practice, community health center, or hospital) for 3,376 physicians with valid National Physician Identifier numbers who went-live on an EHR between July 2007 and December 2011.

Additional data on physician and practice characteristics for PCIP and non-PCIP physicians were derived from the 2011 IMS Health Healthcare Organization Services and Healthcare Relational Services datasets. These files contain physician and practice information, including physician affiliations with practices, for virtually all physicians and practices in the United States. We merged the quality data and PCIP member files with the IMS Health files by National Physician Identifier to include data on physician degree (MD or osteopath), the number of physicians in the practice, and whether a practice had a corporate parent. We also integrated data from the 2000 Decennial Census on the proportion of the population living in poverty (at the zip code level) and whether the practice zip code was located in an urban or rural area. The resulting file contained 5,005 unique physicians (360 from PCIP) and 57,558 physician-measure observations (3,924 from PCIP physicians) representing quality of care for a total of 4,874,518 eligible patient measures (including 271,762 eligible patient measures from PCIP physicians).

Creating a matched comparison group to assess the effect of PCIP on quality

Physicians in PCIP tended to work in smaller practices, were located in higher poverty areas, were less likely to have a corporate parent, and were exclusively located in New York City (Exhibit 1). As a result, PCIP physicians may be expected to have a different quality improvement trajectory than other physicians. To address this issue, we evaluated the effect of PCIP against a matched cohort of non-PCIP physicians. We used a one-to-one matching strategy using propensity scores, matching PCIP physicians to non-PCIP physicians on the number of physicians in their practice, whether the practice had a corporate parent (yes/no), physician degree (MD/DO), physician specialty (internal medicine, family practice, pediatric medicine, cardiology, pulmonology, and gastroenterology), and the proportion of residents living in poverty in the practice zip code. Because all PCIP physicians practiced in urban areas, we limited potential matches to comparison physicians who also practiced in urban areas.

Study Outcome

Our study outcome is outpatient quality of care, defined by the proportion of patients (numerator divided by denominator, or “score”) receiving the ten available process-of-care measures: appropriate treatment for children with upper respiratory infection, appropriate testing for children with pharyngitis, breast cancer screening for women, retinal exam for diabetic patients, HbA1c testing for diabetic patients, cholesterol testing for diabetic patients, urine testing for proteinuria in diabetic patients, cervical cancer screening for women, chlamydia screening for women, and colorectal cancer screening (Exhibit 2).

Exhibit 2.

Physician-level quality scores on quality measures and counts of physicians reporting for PCIP and matched comparison physicians, 2007 and 2010

Measure	Number of observations, 2007-2010	Mean Score (SD), 2007	Mean Score (SD), 2010
Appropriate treatment for children with upper respiratory infection	539	91.4 (8.9)	93.5 (10.3)
Appropriate testing for children with pharyngitis	132	65.8 (34.0)	85.5 (24.1)
Breast cancer screening for women†	1,638	59.4 (11.3)	57.4 (15.0)
Retinal exam for diabetic patients†	590	44.1 (12.6)	44.6 (11.2)
HbA1c testing for diabetic patients	590	71.3 (20.7)	85.3 (10.2)
Cholesterol testing for diabetic patients	591	73.3 (20.8)	85.6 (10.8)
Urine testing for diabetic patients†	590	65.8 (13.9)	69.7 (17.5)
Cervical cancer screening for women	1,933	64.3 (11.4)	77.3 (9.8)
Chlamydia screening for women†	108	42.4 (20.2)	65.0 (19.1)
Colorectal cancer screening†	911	-	53.9 (15.2)

SOURCE: Authors’ analysis

^†Note 1: Denotes EHR sensitive measure

Note 2: Colorectal cancer screening was not available as a quality measure in 2007

Note 3: Differences between the 2007 and 2010 scores are significant at p<.05 for all measures with data in both years

Analysis

We perform a physician-measure-level analysis, in which quality for each measure reported for each physician is an observation in the data, allowing us to evaluate the effect of PCIP on overall quality. We specify a longitudinal fixed-effects model to estimate the effect of varying exposure times to PCIP (i.e. time since going live on the EHR) on quality while controlling for the specific measures reported by physicians. By including physician fixed-effects, we account for all time-invariant factors at the physician level (e.g., physician skill, motivation to improve quality, and patient mix), allowing us to test whether exposure to PCIP improves quality “within” physicians, over time. By controlling for the quality measures reported for each physician, our estimates are invariant to the specific measures reported by physicians.

Physicians in PCIP did not all start the program at the same time. Instead, there was rolling admission to PCIP for the entire study period. Consequently, at the end of any given calendar year (the interval for which quality measures are calculated), exposure to PCIP varied across the cohort of PCIP physicians. To address this, we estimated the continuous, nonlinear effect of each additional month of exposure to PCIP on quality while controlling for secular trends in quality for each study year. Using the model coefficients, we estimated the effects of 6 months, 12 months, 18 months, and 24 months of exposure to PCIP on quality of care.

We extend our base specification by allowing the effect of PCIP to vary depending on the number of technical assistance visits that a physician received. This is performed by including in our model a non-linear interaction between the cumulative number of technical assistance visits that a physician received in a given year and the number of months the physician was exposed to PCIP. We then estimated the effect of 6 months, 12 months, 18 months, and 24 months of exposure to PCIP for physicians associated with practices that received zero technical assistance visits, those who received three technical assistance visits (the median), and those who received eight technical assistance visits (the 90^th percentile).

We also identified quality measures for which previous research has shown a relationship between EHR implementation and quality,(7) and evaluated the effect of EHR implementation only for these “EHR sensitive” measures (breast cancer screening for women, retinal exam for diabetics, urine testing for diabetics, chlamydia screening for women, and colorectal screening). Appendix A provides details on the model estimation and shows the results from several sensitivity checks.

Limitations

We were only able to evaluate the effect of PCIP among the subset of PCIP physicians that matched to the New York Quality Alliance database and met the other study eligibility criteria: our analytic sample included 360 PCIP physicians out of a total of 1,151 small practice physicians in PCIP, 31.3% of participating physicians. PCIP physicians that did not match to the measures file may have been in practices with fewer patients overall, or had fewer patients who were members of health plans contributing to the measures dataset, and therefore did not meet the 30 patient threshold required for any measure to be included in the measures dataset. The inclusion of a greater number of PCIP physicians in our analysis would have allowed for greater statistical power to detect program effects and would have given greater confidence that the results were representative of the experience of all small practice physicians in PCIP, although additional analysis suggests that matched and non-matched PCIP physicians were similar (Appendix A).

We also lacked information on physicians’ payer mix, which would have allowed for a better match between PCIP and comparison physicians. Furthermore, we were limited to evaluating the effect of PCIP for the relatively small set of available measures. PCIP’s technical assistance emphasized improving the quality of care for adult patients with cardiovascular issues, and therefore would not be expected to have uniform effects across the study measures. Our analysis showed some sensitivity of the results to the measures included in the analysis; evaluating the effect of PCIP for a different set of measures may have led to different conclusions. Data used in these analyses were based on paid claims for patients assigned to providers. The difference in quality measurement using a claims-based data source may result in different patterns of quality trends for providers than data derived directly from the practice.(27)

Physicians in PCIP voluntarily joined the program, and the receipt of technical assistance was also voluntary, both of which potentially signal a greater motivation to improve practice processes and quality of care. It is therefore possible that PCIP physicians were, in ways we could not measure, more likely to improve quality than the physicians to whom they were compared. Sensitivity results using all comparison physicians, rather than just the matched sample, found a pattern of similar results, albeit with stronger evidence of the effect of PCIP exposure on quality (Appendix A). However, additional analysis – showing similar trends in quality for PCIP physicians before they joined the program and matched comparison physicians - indicates that the matched comparison physicians appear to be a more appropriate comparison group.

Finally, New York State has made major investments in health information technology through the HEAL New York Phase 10 program. As a result, some of the physicians included in the study who were not part of PCIP may have received support for implementing EHRs that was similar to the support received by physicians who participated in PCIP. Evaluating PCIP against a benchmark of physicians participating in similar programs would likely bias estimated program effects towards the null. Unfortunately, information on whether non-PCIP providers were involved in other implementation projects was not available in this study.

Results

Exhibit 1 shows that, compared to the population of comparison physicians, PCIP physicians were more likely to be located in New York City (100% versus 43.7%), to practice in areas with a higher proportion of residents living in poverty (20.1% versus 13.6%), and to work in smaller practices (2.1 physicians versus 5.4 physicians). They were somewhat less likely to have a specialty in family medicine (15.8% versus 21.4%) but somewhat more likely to have a specialty in pediatric medicine (19.2% versus 16.4%), and were much less likely to have a corporate parent (6.7% versus 26.0%). However, in the matched sample, PCIP and comparison physicians share similar characteristics, with region of practice being the only significant difference between the groups.

Exhibit 2 shows the quality scores and the number of physicians with reported quality for each quality measure for 2007 and 2010. Quality of care was highest for appropriate treatment for children with upper respiratory infection (mean score 91.4 in 2007) and lowest for chlamydia screening for women (mean score 42.4 in 2007). Between 2007 and 2010, the cervical cancer screening for women measure had the most physician-level observations (1,933) while the chlamydia screening for women measure had the fewest number of physician-level observations (108).

Exhibit 3 shows the results from the models estimating the effect of PCIP on quality of care. The column to the left indicates exposure to PCIP (6 months, 12 months, 18 months, and 24 months) corresponding with each estimate. Estimates are shown separately for all quality measures and for the EHR sensitive measures and are shown overall, and for three levels of technical assistance visits (zero visits, three visits, and eight visits). Estimates are interpreted as the percentage point change in quality of care associated with a given period of exposure to PCIP and a given number of technical assistance visits, compared to no exposure to PCIP and no technical assistance. All estimates control for the specific measures reported by PCIP and comparison physicians.

EXHIBIT 3.

Estimates of the effect of PCIP on quality for 6, 12, 18, and 24 months of exposure to PCIP across levels of technical assistance

	Quality measures
	All quality measures				EHR sensitive quality measures
	Cumulative technical assistance visits				Cumulative technical assistance visits
Exposure period	Overall	none	3 visits	8 visits	Overall	none	3 visits	8 visits
6 months	−0.24 (0.53)	−0.30 (0.77)	−0.33 (0.54)	0.04 (0.77)	0.73 (0.66)	0.00 (0.90)	0.74 (0.71)	1.85 (1.05)
12 months	−0.25 (0.72)	−0.65 (1.01)	−0.40 (0.73)	0.25 (0.91)	1.24 (0.88)	−0.09 (1.16)	1.01 (0.90)	2.70* (1.22)
18 months	−0.02 (0.81)	−0.86 (1.05)	−0.23 (0.82)	0.61 (0.90)	1.71 (0.95)	0.07 (1.22)	1.13 (0.94)	2.90* (1.15)
24 months	0.42 (0.91)	−0.74 (1.15)	0.14 (0.94)	1.10 (1.07)	2.31* (1.05)	0.81 (1.33)	1.41 (1.06)	2.79* (1.29)
Observations	7,622				3,589
Number of total physicians	720				516
Number of PCIP physicians	360				258

SOURCE: Authors’ analysis

^*Note 1: p<.05

Note 2: 3 cumulative technical assistance visits is the median number of visits and 8 cumulative visits is the 90^th percentile among practices that went live during the study period.

Note 3: Effects are interpreted as the incremental percentage point change in quality of care for a given exposure to PCIP

Note 4: Standard errors robust to groupwise heterskedasticity at the level of the match shown in ()

Note 5: EHR sensitive measures include breast cancer screening for women, retinal exam for diabetics, urine testing for diabetics, chlamydia screening for women, and colorectal screening

Exhibit 3 shows that, for all quality measures, PCIP was not significantly associated with improvement in quality of care at 6 months (−0.24 percentage points, p > .10), 12 months (−0.25 percentage points p > .10), 18 months (−0.02 percentage points p > .10), or 24 months of exposure (+0.42 percentage points p > .10). In addition, although effects were positive for practices receiving 8 technical assistance visits, no level of exposure to PCIP was significantly associated with an increase in performance for all quality measures for any level of technical assistance.

Exhibit 3 also shows results for EHR sensitive measures. While lower exposure periods to PCIP were not associated with quality improvements for EHR sensitive measures, 24 months of exposure to PCIP was significantly associated with quality improvement (+2.31 percentage points, p < .05). This effect was driven by physicians in practices that received extensive technical assistance: while 24 months of program exposure was not associated with quality improvement for physicians in practices that received zero technical assistance visits (+0.81 percentage points, p > .10) or three technical assistance visits (+1.41 percentage points, p > .10), 24 months of exposure was significantly associated with quality improvement for physicians in practices that received eight technical assistance visits (+2.79 percentage points, p < .05).

Exhibit 4 shows monthly estimates of exposure to PCIP on quality for the EHR sensitive measures, derived from the estimated models. Exposure to PCIP was not associated with quality improvement for physicians in practices receiving zero or three quality visits. However, for those receiving at least eight quality visits, 9 through 24 months of exposure to PCIP was significantly associated with greater quality improvement (p <.05).

EXHIBIT 4.

Caption/headline: Estimates of the effect of exposure to the Primary Care Information Project on quality across levels of technical assistance for EHR sensitive measures

SOURCE: Authors’ analysis

Note 1: EHR sensitive measures include breast cancer screening for women, retinal exam for diabetics, urine testing for diabetics, chlamydia screening for women, and colorectal screening

Discussion

This study used a unique dataset based on medical claims from multiple payers in New York State to evaluate the effect of the Primary Care Information Project, a large electronic health record implementation and technical assistance program aimed at small primary care practices serving disadvantaged populations, on ten commonly used measures of outpatient quality of care. Within the first two years of the program, use of an EHR was not associated with an overall improvement in quality, even among physicians who received high levels of technical assistance from PCIP. However, for a limited set of quality measures that previous research has shown to be sensitive to EHR implementation, we found that use of the EHR was associated with significantly higher quality of care for physicians who received high levels of technical assistance (eight or more visits) and who had been using the EHR for at least nine months. Exposure to PCIP was not associated with quality improvement for physicians who received low or moderate levels of technical assistance. These findings suggest that the early effects of a major electronic health record implementation project, as measured by claims-based measures, were limited to a subset of quality measures, and limited to physicians in practices receiving extensive technical assistance from the program.

Our results are consistent with other research finding that EHRs alone do not consistently improve quality of care.(8) However, ours is one of the few studies evaluating the effect of EHR implementation on quality of care in a community, outpatient setting: most studies have been single institution studies, many of which were conducted within institutions recognized as health IT leaders.(5, 6, 13) Our study focuses on physicians who are a key target for regional extension centers – physicians in small practices who serve primarily disadvantaged patients.

Our results support three main conclusions. First, EHR implementation through the PCIP program was associated with an improvement in quality for a subset of quality measures which previous research has shown to be related to quality of care (breast cancer screening for women, retinal exam for diabetics, urine testing for diabetics, chlamydia screening for women, and colorectal screening). EHR implementation and technical assistance, in the short term, may not help physicians improve quality across all clinical parameters.

Second, EHR implementation alone was not sufficient to improve quality of care. Only those physicians who received high levels of technical assistance concomitant with EHR implementation improved quality. Even relatively long periods of EHR use – up to two years – were not associated with quality improvement for physicians who received no technical assistance, or moderate levels of technical assistance. For the quality measures we assessed, this finding stands in contrast to widespread physician perceptions that EHRs improve quality of care, (28) and suggests that Regional Extension Centers have an important role to play in facilitating the use of EHRs to improve quality of care.

Third, even with high levels of technical assistance, it took close to one year of exposure to PCIP for effects on quality of care to be observed. This suggests that there is a learning curve for using the EHR effectively to improve quality, and that it is important for future studies to appropriately capture the effect of EHRs on quality by evaluating medium to longer term outcomes. EHR implementation is disruptive to physician practices; it is possible that it takes longer than two years for improvements in quality to appear.

Conclusion

Our study suggests that small primary care practices serving disadvantaged populations can use EHRs to improve the quality of care they provide. However, improvements were small, were only for a limited number of measures, and occurred after at least nine months of using the EHR in practices that received eight or more technical assistance visits. These findings suggest that small practices in disadvantaged areas may need considerable help to use EHRs to improve quality, and lend some support for the Regional Extension Center model of assisting practices use EHRs.(29, 30) However, it should be noted that the practices in this study received assistance from PCIP, an experienced and well-funded organization that also provided funding to assist practices obtain appropriate software. It will be important to compare the effectiveness of different regional extension centers to each other, and to evaluate these effects over the long term.