Provider and patient satisfaction with the integration of ambulatory and hospital EHR systems

Abstract

Objective

The installation of EHR systems can disrupt operations at clinical practice sites, but also lead to improvements in information availability. We examined how the installation of an ambulatory EHR at OB/GYN practices and its subsequent interface with an inpatient perinatal EHR affected providers’ satisfaction with the transmission of clinical information and patients’ ratings of their care experience.

Methods

We collected data on provider satisfaction through 4 survey rounds during the phased implementation of the EHR. Data on patient satisfaction were drawn from Press Ganey surveys issued by the healthcare network through a standard process. Using multivariable models, we determined how provider satisfaction with information transmission and patient satisfaction with their care experience changed as the EHR system allowed greater information flow between OB/GYN practices and the hospital.

Results

Outpatient OB/GYN providers became more satisfied with their access to information from the inpatient perinatal triage unit once system capabilities included automatic data flow from triage back to the OB/GYN offices. Yet physicians were generally less satisfied with how the EHR affected their work processes than other clinical and non-clinical staff. Patient satisfaction dropped after initial EHR installation, and we find no evidence of increased satisfaction linked to system integration.

Conclusions

Dissatisfaction of providers with an EHR system and difficulties incorporating EHR technology into patient care may negatively impact patient satisfaction. Care must be taken during EHR implementations to maintain good communication with patients while satisfying documentation requirements.

Introduction

Despite widespread belief among policy makers that electronic health records (EHRs) can improve the quality of care and patient satisfaction, research on provider and patient satisfaction with current EHR technology is mixed.^1,2 While some providers agree that an EHR can improve care, many are concerned about its effects on their work processes and interactions with patients.^3–8 Likewise, while some studies find that patient satisfaction improves with the use of EHRs, others conclude that patients are often unhappy with their exam room experience.^9–14 However, researchers have not examined how the integration of EHRs located in different care settings affects provider and patient satisfaction. Such integration has been highlighted as a means of reducing information loss and duplication of care, which could mitigate some of the negative perceptions of EHRs reported in the past.¹⁵

We study provider and patient satisfaction during the adoption of a commercial ambulatory EHR in obstetrics and gynecology (OB/GYN) practices, and its subsequent integration with the hospital’s perinatal EHR, at the Lehigh Valley Health Network (LVHN) in eastern Pennsylvania. We found in previous studies of this implementation that two-way data sharing between practice sites and the hospital increased care quality for OB patients,¹⁶ but that the necessary changes to practice site work processes decreased productivity.^16,17 We now examine the impact of integration on provider satisfaction with information availability and with the EHR system. We also investigate how EHR integration affects patient satisfaction with their care experience, first for the general OB/GYN patient population and then for OB patients, who directly benefit from shared information.

Obstetric care is naturally fragmented because it involves multiple providers and health care resources over a period of several months, making it more likely that integration will improve care. For example, an LVHN obstetrics patient normally visits her OB/GYN office 12 to 13 times during her pregnancy, and typically does not see the same provider at each visit. Further, approximately 10% of pregnant LVHN patients visit the hospital’s perinatal triage unit during weeks 20 to 36 of their pregnancy, and about 50% visit during weeks 37 to 41 for a “labor check.” Patients not admitted to the hospital after a visit to triage continue office appointments until a subsequent visit to triage results in their admission to the hospital, or until an outpatient physician directly admits them. Ideally, information indicating that a patient visited triage, and the results of tests and assessments done there, would be available to physicians during subsequent office visits. However, physicians in primary-care offices frequently lack discharge summaries from patients’ hospital visits; this was true at LVHN prior to their adoption of an integrated EHR, where the transmission of obstetric patient data was costly and slow.^18–21

In 2009 LVHN installed a commercial EHR at the OB/GYN practices which was subsequently integrated with the EHR system at the hospital triage unit. The new system improved information transmission, but also reduced physician productivity for a significant period of time and required changes to work processes.^16,22 We study whether these significant changes to the provider workplace were viewed positively or negatively by physicians and both clinical and non-clinical staff at two large OB/GYN practice groups, as well as how these changes affected patients’ satisfaction.

Patient satisfaction survey results have received greater scrutiny in recent years, in part because Medicaid payments are now tied directly to measures of patient satisfaction under the Hospital Value-Based Purchasing Program.²³ Private insurers are also expected to increasingly tie payments to measures of quality, such as patient satisfaction, in the future.²⁴

Methods

Implementation of the EHR system at LVHN occurred in 3 stages, and we surveyed providers at 4 discrete points during the process. In contrast, patient satisfaction surveys were collected continually. See Figure 1 for descriptions and timings of the stages and for survey response counts. Note that round 1 of the provider survey received fewer responses than subsequent rounds. This is because the initial provider surveys were administered through email, whereas subsequent surveys were administered at staff meetings. We verified that respondents to the initial survey were not disproportionately from a particular group by comparing respondent characteristics in survey rounds 2-4 to the initial round (Supplementary Appendix Table A1).

Figure 1.

EHR Implementation timeline and survey implementation.

Provider Satisfaction

The provider survey asked about both the availability of data from triage visits and the respondent’s satisfaction with the EHR system, and we analyze these responses separately. Survey responses could range from 1 (disagree strongly) to 5 (agree strongly). Although the survey response categories are ordinal, we model them as a cardinal index ranging from 1 to 5 in order to increase the statistical power of our estimates. Specifically, we use a zero-truncated Poisson (ZTP) model where the condition mean of the index is specified as $\mathrm{E}\left(Y|Y>0\right)=\mu +\sigma \mathrm{}\mathrm{w}\mathrm{i}\mathrm{t}\mathrm{h}\mathrm{\sigma }=\mu e-1$.²⁵ When we analyze responses,$Y$, by the clinical staff member i in year t at OB/GYN practice sites to questions about the availability of information from triage (the questions are on Table 1), we specify the rate parameter in the Poisson model as:

$${\mu }_{it}=\mathrm{}\mathrm{e}\mathrm{x}\mathrm{p}({\beta }_0+{\beta }_1·{2010}_t+{\beta }_2·{2012}_t+\mathrm{}{\beta }_3·{2013}_t\mathrm{}+{\beta }_4·C_{it}),$$ (1)

where 3 indicator variables, 2010, 2012, and 2013, represent the year of the survey; and $C\mathrm{}$is a vector of provider characteristics.

Table 1.

Effect of EHR Integration on the Availability of Information from Triage to Clinical Staff at OB/GYN Offices^a

	Marginal Effects from Zero-Truncated Poisson Model
Survey Questions	2010	2012	2013
#1 Information that I need from visits to triage is complete.	−0.625	0.498	0.844*
	(0.217)	(0.369)	(0.064)
	[0.579]	[0.422]	[0.102]
#2 Information that I need from visits to triage is easily accessible.	−0.864	0.684	1.079**
	(0.102)	(0.264)	(0.033)
	[0.408]	[0.352]	[0.066]
#3 Documentation from visits to triage is incorporated into the office prenatal record.	0.614	3.007**	3.441***
	(0.507)	(0.008)	(0.001)
	[0.579]	[0.032]	[0.004]
#4 Laboratory tests and diagnostic studies performed at triage are recorded in the office prenatal record.	−0.048	0.905	0.560
	(0.927)	(0.131)	(0.310)
	[0.927]	[0.210]	[0.354]
#5 New diagnoses determined on triage are recorded in the office prenatal problem lists within one week.	−0.315	0.171	0.185
	(0.487)	(0.647)	(0.633)
	[0.579]	[0.647]	[0.633]
#6 Lack of information from visits to triage makes it difficult to manage patients.	−0.215	−0.885***	−0.823***
	(0.444)	(0.002)	(0.000)
	[0.579]	[0.016]	[0.000]
#7 Errors are made with the potential to harm patients because records from triage are not available.	−0.230	−0.819**	−0.566**
	(0.420)	(0.013)	(0.026)
	[0.579]	[0.035]	[0.066]
#8 It is easier to re-order diagnostic and laboratory tests in the office than check whether they were performed on triage.	1.230*	0.962	1.106*
	(0.071)	(0.122)	(0.092)
	[0.408]	[0.209]	[0.123]

^aUnadjusted p-values are in parentheses based on standard errors that are clustered at the person-level; “family-wise” p-values are in brackets. All regressions include respondent’s age, years of experience, years of experience at current practice site, whether they are a physician (MD or DO), whether the respondent works at practice 2, and an indicator for missing provider characteristics. Sample size is 108 clinical staff for all questions except for #1, which has 107 responses.

^***p < .01, **p < .05, *p < .1 (based on unadjusted p-values).

The year indicators are the key variables in equation (1): 2010_t equals 1 if the response was from the second survey, 2012_t equals 1 if the response was from the third survey, and 2013_t equals 1 if the response was from the fourth survey. The provider characteristics in $C$ include age, years of experience, years of experience at the current practice site location, whether the respondent is a physician (MD or DO), and whether the respondent works at Practice B (as opposed to Practice A). If data on characteristics are missing, we assign the sample average of the characteristic and include a variable that equals 1 to indicate that the observation has assigned values for one or more characteristics.

We next analyze responses to questions about provider satisfaction that we added to the survey over time in response to in-depth interviews with providers carried out for a matching qualitative study.¹⁷ These questions are reported in Table 2. We were most interested in systematic differences in the responses of physicians and non-physician clinicians, as studies of EHRs in primary care offices suggest that assessments differ according to provider type.^26,27 We therefore modified equation (1) to include 2 additional variables, one of which equals 1 if the respondent was a physician (MD or DO) and the other equals 1 if the respondent was other clinical staff (certified nurse midwife, certified registered nurse practitioner, physician’s assistant, or registered nurse), leaving non-clinical staff as the reference group. We also modified the set of year indicators in equation (1) based on when each question was added to the survey.

Table 2.

Differences Between Physician and Non-physician Responses to EHR Assessment^a

	Marginal Effects from Zero-Truncated Poisson Model
Survey Questions	Physician	Nb
#9 The ambulatory EHR is easy to use.	−1.666***	271
	(0.000)
	[0.000]
#10 Someone is there to assist me when I have system difficulties.	−0.911***	258
	(0.000)
	[0.000]
#11 Using the ambulatory EHR increases my productivity.	−1.800***	258
	(0.000)
	[0.000]
#12 I find the ambulatory EHR useful.	−1.310***	176
	(0.000)
	[0.000]
#13 Using the EHR improves my job performance.	−1.532***	176
	(0.000)
	[0.000]
#14 Using the ambulatory EHR enhances my effectiveness on the job.	−1.481***	176
	(0.000)
	[0.000]
#15 All things considered, my continuing use of the ambulatory EHR is beneficial.	−1.324***	176
	(0.000)
	[0.000]

^aUnadjusted p-values are in parentheses based on standard errors that are clustered at the person-level; “family-wise” p-values are in brackets. The sample includes responses from both clinical and non-clinical providers. All regressions include respondent’s age, years of experience, years of experience at current practice site, an indicator for non-physician clinical staff, whether the respondent works at practice 2, an indicator for missing provider characteristics, and survey year dummies.

^bQuestion #9 was asked in all 4 surveys, questions #10 and #11 were on the last 3 surveys (2010, 2012, 2013), and questions #12-15 were on the last 2 surveys (2012, 2013).

^***p < .01, **p < .05, *p < .1 (based on unadjusted p-values).

Descriptive statistics for the survey questions and the independent variables for these analyses are shown on Supplementary Appendix Tables A2 and A3.

Patient Satisfaction

A representative sample of patients seen at the OB/GYN offices from January 2007 through December 2012 received a Press-Ganey survey asking them to rate their experience. Responses could range from 1 (very poor) to 5 (very good). Over 8000 surveys were returned for an approximate response rate of 25% (not all respondents answered all questions). We restricted our analysis to questions that were asked in all the surveys during the sample period, and for which we had at least 3900 responses (the questions are on Table 3).

Table 3.

Effect of EHR Integration and Learning on the Satisfaction of Obstetric and Gynecologic Patients^a

Survey questions	Stage_1	Stage_2	Stage_3	Recoverb	N
#1 Friendliness/courtesy of nurse/asst	−0.137***	−0.083**	0.053	9, c	8026
	(0.000)	(0.035)	(0.343)
	[0.003]	[0.171]	[0.947]
#2 Concern of nurse/asst for problem	−0.154***	−0.083**	0.019	9, c	7381
	(0.001)	(0.046)	(0.768)
	[0.008]	[0.171]	[0.947]
#3 Friendliness/courtesy of care provider	−0.053	−0.058**	0.006	14, c	8021
	(0.122)	(0.082)	(0.912)
	[0.151]	[0.171]	[0.947]
#4 Care provider explanations of prob/condition	−0.077*	−0.076**	0.008	9, c	7795
	(0.079)	(0.056)	(0.892)
	[0.103]	[0.171]	[0.947]
#5 Care provider concern for questions/worries	−0.098**	−0.099**	−0.019	12, c	7860
	(0.035)	(0.015)	(0.763)
	[0.053]	[0.150]	[0.947]
#6 Care provider efforts to include in decisions	−0.092**	−0.044	0.066	9, c	7349
	(0.043)	(0.289)	(0.312)
	[0.059]	[0.376]	[0.947]
#7 Care provider information about medications	−0.109**	−0.113**	−0.059	11, c	5880
	(0.029)	(0.017)	(0.395)
	[0.050]	[0.150]	[0.947]
#8 Care provider instructions for follow-up care	−0.144***	−0.088*	−0.025	13, c	6523
	(0.004)	(0.063)	(0.706)
	[0.015]	[0.171]	[0.947]
#9 Care provider spoke using clear language	−0.083**	−0.061*	0.018	9, c	7904
	(0.021)	(0.075)	(0.710)
	[0.038]	[0.171]	[0.947]
#10 Time care provider spent with patient	−0.147***	−0.066	0.042	10, c	7981
	(0.003)	(0.130)	(0.504)
	[0.015]	[0.189]	[0.947]
#11 Patients' confidence in care provider	−0.053	−0.064	0.016	10, c	7991
	(0.228)	(0.129)	(0.818)
	[0.247]	[0.189]	[0.947]
#12 Likelihood of recommending provider	−0.069	−0.075	0.059	11, c	7957
	(0.153)	(0.107)	(0.413)
	[0.173]	[0.185]	[0.947]
#13 Convenience of our office hours	−0.039	−0.062	0.005	4	8004
	(0.376)	(0.131)	(0.947)
	[0.376]	[0.189]	[0.947]
#14 Our sensitivity to patients' needs	−0.138***	−0.025	0.070	6	7895
	(0.003)	(0.528)	(0.245)
	[0.015]	[0.572]	[0.947]
#15 Our concern for patients' privacy	−0.095***	−0.042	−0.016	8	7910
	(0.008)	(0.236)	(0.747)
	[0.027]	[0.322]	[0.947]
#16 Staff respected having family with them	−0.050	0.010	0.075	6	3977
	(0.314)	(0.806)	(0.154)
	[0.327]	[0.806]	[0.947]
#17 Cheerfulness of practice	−0.1194***	−0.066*	−0.067	6	7997
	(0.004)	(0.068)	(0.219)
	[0.015]	[0.171]	[0.947]
#18 Cleanliness of our practice	−0.084**	0.014	−0.004	7	7981
	(0.011)	(0.659)	(0.926)
	[0.029]	[0.686]	[0.947]
#19 Care received during visit	−0.102**	−0.060*	−0.030	7	8007
	(0.015)	(0.092)	(0.599)
	[0.033]	[0.171]	[0.947]
#20 Likelihood of recommending practice	−0.108**	−0.064*	0.016	8	7980
	(0.011)	(0.087)	(0.785)
	[0.029]	[0.171]	[0.947]
#21 Ease of scheduling appointments	−0.106**	−0.029	0.022	4	8021
	(0.039)	(0.526)	(0.792)
	[0.056]	[0.572]	[0.947]
#22 Courtesy of person scheduling apt	−0.062	−0.068*	−0.019	4	8005
	(0.143)	(0.088)	(0.778)
	[0.169]	[0.171]	[0.947]
#23 Our promptness in returning calls	−0.124**	−0.130**	−0.182**	7	5830
	(0.0345)	(0.014)	(0.021)
	[0.053]	[0.150]	[0.548]
#24 Ability to get desired appointment	−0.301***	−0.048	0.076	8, - in 36, d	7953
	(0.000)	(0.393)	(0.475)
	[0.000]	[0.487]	[0.947]
#25 Ease of getting clinic on phone	−0.131**	−0.085	−0.127	3	7163
	(0.017)	(0.071)	(0.070)
	[0.034]	[0.171]	[0.912]
#26 Courtesy of registration staff	−0.106**	−0.030	−0.047	5	8018
	(0.015)	(0.424)	(0.411)
	[0.033]	[0.501]	[0.947]

^aUnadjusted p-values are in parentheses based on standard errors that are clustered at the patient level; “family-wise” p-values are in brackets. All regressions include control variables for patient characteristics, experiences on the day of the visit; an indicator for a missing risk score, and indicators for practice, practice site, quarter, three-digit patient zip code, and year.

^bMonths until satisfaction recovers to original level.

^cSatisfaction recovers in the months shown, but becomes negative after Stage 2, then recovers after Stage 3.

^dSatisfaction recovers, but becomes negative after 36 months.

^***p < .01, **p < .05, *p < .1 (based on unadjusted p-values).

Again, we analyze the response,$Y,$ by patient i, treated at practice site k, at time t as a cardinal index ranging from 1 to 5 using the ZTP model. In this case the rate parameter is specified as:

$${\mu }_{ikt}=\mathrm{}{\mathrm{e}\mathrm{x}\mathrm{p}(\beta }_0+{\beta }_1·Stag{e\_1}_{kt}+{\beta }_2·Stag{e\_2}_{kt}+{\beta }_3·Stag{e\_3}_{kt}+{\beta }_4·L_{kt}+\mathrm{}{\beta }_5·P_{it}\mathrm{}+{\beta }_6{·R}_{kt})$$ (2)

where $Stage\_1_{kt}$, $Stage\_2_{kt}$, and $Stage\_3_{kt}$ are variables indicating the stage of EHR installation and interface achieved at practice site k at time t;$L$ is a vector of two variables included to measure learning by providers in practice site k;$P$ is a vector of patient characteristics and experiences on the day of her visit; and $R$ is a vector of variables indicating practice site and time. The Stage_1_kt indicator in equation (2) measures the impact of EHR installation relative to the pre-installation period, and the $Stage\_2_{kt}$ (partial EHR integration) and $Stage\_3_{kt}$ (full EHR integration) indicators each measure the impact of that stage of integration relative to Stage 1.

We model learning by providers as a quadratic function, using the number of months since installation of the ambulatory EHR at the office the patient visited, and the number of months squared. The learning variables capture not only the impact of changes in provider facility with the EHR on patient satisfaction, but also the impact of changes in clinical practice patterns caused by providers’ greater knowledge of EHR capabilities.

We include variables to control for the effects of the responding patient’s age, race/ethnicity, insurance type, and prior health risk. These include categorical variables for age (less than 18, 18-34, 35-44, and over 44) race/ethnicity (white and nonwhite), insurance type (Medicaid, Medicare, private insurance, and uninsured), and a variable indicating coverage through a private or public managed care plan. The location of the patient’s residence is captured by eleven indicator variables for 3-digit zip code, and variables indicating residence in New Jersey or any other state.

We generate risk scores to control for a respondent’s pre-existing conditions and overall illness severity using the Diagnosis Cost Groups/Hierarchical Condition Categories (DCG/HCC) method. DCG/HCC risk scores are derived from data on patient age, sex, and physician-reported diagnosis codes (ICD-9-CM). They have been validated as a proper measure of risk adjustment in both inpatient and outpatient settings.^28–30 We calculated risk scores using diagnosis codes reported in LVHN billing databases and include variables to indicate membership in the top three quartiles of the risk score distribution. We also include a variable to indicate that a risk score could not be calculated for the patient.

We include variables indicating whether the patient was making her first visit to the office, and categorical variables to measure wait times in the waiting room and in the exam room 0-10, 10-30, 30-60, or more than 60 minutes). We include wait times to control for their likely strong effects on satisfaction. However, the coefficients of these variables may capture some of the effects of the EHR system if, for example, the new system unexpectedly increased visit lengths, and wait times as a result. We therefore estimated our specifications without these variables, but obtained qualitatively similar results.

Finally, we include indicator variables for membership in the top three quartiles of the distribution of how many days it took the patient to return the survey, a variable measuring whether the survey was administered in English or Spanish, variables indicate the credentials of the care provider (medical doctor, doctor of osteopathic medicine, certified nurse midwife, certified registered nurse practitioner, registered nurse, and physician’s assistant), indicator variables for the specific practice sites affiliated with practice groups A and B, and indicator variables for quarter and year.

We then re-estimated equation (2) on a subsample of OB patients, who we identified as patients whose visit occurred within 12 months before to 6 months after the delivery of a baby. Descriptive statistics for the survey responses and the independent variables for the full sample and the OB subsample are shown on Supplementary Appendix Tables A4 and A5.

Since the ZTP is a nonlinear model, we report the marginal effects of key independent variables of interest from equations (1) and (2). The marginal effects measure the effect of being in the measured category on the level of satisfaction, as measured by the response index. We cluster the standard errors at the respondent level for equation (1) and at the patient level for equation (2). For all estimations we report both unadjusted p-values and “family-wise” p-values, ie, p-values adjusted to correct for the underlying correlation between responses to related questions on the surveys.^31,32

Results

Estimation results for the provider satisfaction model are reported on Tables 1 and 2, and estimates from the patient satisfaction model are reported on Tables 3 and 4. All tables contain unadjusted p-values in parentheses and family-wise p-values in brackets. Stars indicate significance levels with respect to the unadjusted p-values, but we confine most of our discussion to coefficients with family-wise p-values indicating significance at least at the 10% level.

Table 4.

Effect of EHR Integration and Learning on the Satisfaction of Obstetric Patients Only^a

Survey questions	Stage_1	Stage_2	Stage_3	Recoveryb	N
#1 Friendliness/courtesy of nurse/asst	−0.192**	−0.124*	0.024	9, - c	2335
	(0.022)	(0.068)	(0.790)
	[0.125]	[0.787]	[0.986]
#2 Concern of nurse/asst for problem	−0.097	−0.109	−0.062	8, d	2199
	(0.351)	(0.140)	(0.514)
	[0.351]	[0.787]	[0.973]
#3 Friendliness/courtesy of care provider	−0.121	−0.018	0.138	12, d	2336
	(0.150)	(0.789)	(0.213)
	[0.239]	[0.900]	[0.973]
#4 Care provider explanations of prob/condition	−0.115	−0.021	0.087	10, d	2318
	(0.292)	(0.780)	(0.487)
	[0.316]	[0.900]	[0.973]
#5 Care provider concern for questions/worries	−0.135	−0.023	−0.002	12, d	2321
	(0.234)	(0.757)	(0.986)
	[0.276]	[0.900]	[0.986]
#6 Care provider efforts to include in decisions	−0.134	−0.014	0.181	9, d	2206
	(0.194)	(0.852)	(0.151)
	[0.265]	[0.900]	[0.973]
#7 Care provider information about medications	−0.166	−0.058	0.023	8, d	1697
	(0.160)	(0.481)	(0.830)
	[0.239]	[0.814]	[0.986]
#8 Care provider instructions for follow-up care	−0.232**	−0.029	−0.008	12, d	2012
	(0.036)	(0.717)	(0.940)
	[0.134]	[0.900]	[0.986]
#9 Care provider spoke using clear language	−0.081	−0.046	0.044	6, d	2320
	(0.334)	(0.478)	(0.600)
	[0.348]	[0.814]	[0.973]
#10 Time care provider spent with patient	−0.222*	−0.052	0.068	14, d	2333
	(0.063)	(0.532)	(0.585)
	[0.149]	[0.814]	[0.973]
#11 Patients' confidence in care provider	−0.148	−0.053	0.080	10, d	2333
	(0.166)	(0.510)	(0.531)
	[0.239]	[0.814]	[0.973]
#12 Likelihood of recommending provider	−0.141	−0.015	0.125	9, d	2324
	(0.215)	(0.865)	(0.349)
	[0.271]	[0.900]	[0.973]
#13 Convenience of our office hours	−0.090	−0.079	−0.026	4, - e	2329
	(0.264)	(0.242)	(0.838)
	[0.298]	[0.787]	[0.986]
#14 Our sensitivity to patients' needs	−0.163*	−0.015	0.204*	7	2321
	(0.077)	(0.835)	(0.064)
	[0.154]	[0.900]	[0.750]
#15 Our concern for patients' privacy	−0.138**	−0.000	0.084	9	2320
	(0.035)	(0.900)	(0.345)
	[0.134]	[0.997]	[0.973]
#16 Staff respected having family with	−0.134	0.051	0.199	7	1628
	(0.128)	(0.355)	(0.019)
	[0.221]	[0.814]	[0.486]
#17 Cheerfulness of practice	−0.196**	−0.0667	−0.015	7	2327
	(0.043)	(0.296)	(0.871)
	[0.141]	[0.814]	[0.986]
#18 Cleanliness of our practice	−0.130*	0.071	0.004	7	2325
	(0.059)	(0.199)	(0.961)
	[0.149]	[0.787]	[0.986]
#19 Care received during visit	−0.180*	−0.043	0.037	10	2329
	(0.062)	(0.518)	(0.700)
	[0.149]	[0.814]	[0.986]
#20 Likelihood of recommending practice	−0.163*	−0.045	0.101	9, - e	2327
	(0.076)	(0.497)	(0.357)
	[0.154]	[0.814]	[0.973]
#21 Ease of scheduling appointments	−0.142*	−0.167**	−0.107	7, - c	2334
	(0.096)	(0.017)	(0.368)
	[0.178]	[0.436]	[0.973]
#22 Courtesy of person scheduling apt	−0.116	−0.082	−0.011	7	2336
	(0.219)	(0.237)	(0.931)
	[0.271]	[0.787]	[0.986]
#23 Our promptness in returning calls	−0.332***	−0.113	−0.110	8, - e	1904
	(0.007)	(0.201)	(0.384)
	[0.064]	[0.787]	[0.973]
#24 Ability to get desired appointment	−0.348***	−0.040	0.121	8, - c	2323
	(0.001)	(0.677)	(0.526)
	[0.017]	[0.900]	[0.973]
#25 Ease of getting clinic on phone	−0.237**	−0.115	−0.180*	12	2116
	(0.024)	(0.145)	(0.087)
	[0.125]	[0.787]	[0.750]
#26 Courtesy of registration Staff	−0.229***	−0.046	−0.041	9	2323
	(0.007)	(0.470)	(0.636)
	[0.064]	[0.814]	[0.973]

^aUnadjusted p-values are in parentheses based on standard errors that are clustered at the patient level; “family-wise” p-values are in brackets. All regressions include control variables for patient characteristics, experiences on the day of the visit; an indicator for a missing risk score, and indicators for practice, practice site, quarter, three-digit patient zip code, and year.

^bMonths taken for satisfaction to recover to original level.

^cSatisfaction recovers, but becomes negative after Stage 2.

^dSatisfaction recovers in the months shown, but becomes negative after Stage 2, then recovers after Stage 3.

^eSatisfaction recovers, but becomes negative after Stage 3.

^***p < .01, **p < .05, *p < .1 (based on unadjusted p-values).

Provider Satisfaction

The results in Table 1, where each set of reported marginal effects for survey year is from an estimation of equation (1) for the indicated question, show how perceptions of clinical staff at OB/GYN offices about the availability of information from inpatient triage visits changed as the EHR system was implemented. Information did not flow through the EHR from triage to the OB/GYN offices until Stage 2 was implemented in 2011, and clinical staff responses in 2010 show that the staff perceived no change in the availability of triage information relative to 2009 when the EHR system was installed.

By 2012, however, when the triage summary was automatically sent to the ambulatory EHR, clinical providers agreed that documentation from triage visits was incorporated into the prenatal record (question #3) to a much greater extent than in 2009, and agreed that lack of information from triage visits makes it difficult to manage patients (question #6) or that potentially harmful errors were made due to lack of triage information (questions #7) to a lesser extent than in 2009. By 2013 when there was complete two-way exchange of information, clinical staff agreed that triage information is easily accessible (#2) and incorporated into the office prenatal record (#3) to a greater extent, while they agreed that lack of information about triage visits makes it difficult to manage patients (#6) or is likely to cause harm (#7) to a lesser extent.

Table 2 shows how the satisfaction of physicians with the EHR differs from other clinical staff and non-clinical staff. (There were no statistically significant differences between the responses of other clinical staff and non-clinical staff.) Physicians were uniformly dissatisfied with the system and its implementation compared to other staff members.

Patient Satisfaction

Estimates of the patient satisfaction models from the combined sample of both gynecological and obstetric patients are given in Table 3. The estimated coefficient for Stage_1 is negative for every question, and statistically significantly for 19 of the 26 questions based on the family-wise p-values. Patient satisfaction decreased with virtually all aspects of their experience. Question #20 provides a good overall summary of patient dissatisfaction after Stage 1: the extent to which patients recommend the practice to others drops by 0.1 index points, or 2.3%.

The marginal effect estimates for Stage_2, the variable indicating transmission of a triage visit summary back to the OB/GYN offices, are still almost uniformly negative, but are less precisely estimated. The additional reduction in satisfaction from Stage 2 implementation appears concentrated among the set of questions measuring interactions with the care providers (questions #1-#5, #7, and #8), although none are statistically significant based on the family-wise p-values. Finally, the marginal effects of the Stage_3 variables, while now mostly positive, are also insignificant based on the family-wise p-values, indicating no significant impact on patient satisfaction of the final stage of EHR integration relative to Stage 1.

We investigated the overall impact of the integrated EHR by calculating for each question the number of months it would take for patient satisfaction to recover to its original level given the cumulative impact of all stages of implementation plus the effects of learning. Numbers in the column titled “Recover” indicate that patient satisfaction eventually recovered to pre-EHR level for all but #24, but that the initial recovery in many cases was lost again with the implementation of stage 2, before recovering again after stage 3 (questions #1-12)—all of these questions relate to evaluating the actual care providers.

When we subset to only obstetric patients the estimates of the patient satisfaction model are qualitatively similar to those from the full sample (Table 4). All estimated marginal effects of the Stage 1 indicator remain negative, but many fewer are statistically significant. The exceptions are the strongly negative results for questions #23, #24, and #26, all dealing with practice administrative issues. Likewise, the marginal effect estimates for Stage_2 are still almost uniformly negative, but none are significant, while the estimated coefficients of Stage_3 are more likely to be positive but again appear to have no significant effect on obstetric patient satisfaction. Despite the lack of precision on the smaller sample, the estimates for all three stages generally increase in magnitude.

Examination of the time to recover, however, reveals that while satisfaction of the OB patients with care providers recovered as or more quickly than the satisfaction of the entire sample (#2-#12), their satisfaction was slower to recover with respect to other aspects of the practices. In some cases, after an initial recovery, patient satisfaction fell again and never returned to their original levels. This occurred after stage 2 for questions #1, #21, and #24, and after stage 3 for questions #13, #20, and #23. Dissatisfaction thus appears most persistent with respect to administrative aspects of the practices.

Discussion and Conclusions

We study provider and patient satisfaction during installation of an ambulatory EHR and its subsequent integration with the inpatient EHR at the hospital’s triage subunit. We find that clinical provider satisfaction with the availability of information improved following Stages 2 and 3, when information on triage visits was sent automatically to the ambulatory EHR. For example, one physician remarked during an interview: “The fact that the document [Triage Summary] shows up in an inbox is a huge plus,” enabling the practice to follow up with patients after triage visits; previously, he had often not even known about a triage visit unless the patient told him. Physicians also noted that the discrete triage data going into the office flow sheet improved their next interaction with that patient.

Yet physicians in particular were the most dissatisfied with the EHR, as the benefits of better information did not compensate for the negative effects of the system on their work. Physicians had new documentation requirements and reported navigation difficulties, while other clinical staff more quickly found efficiency improvements through shared information, such as lab results.¹⁷ This result is consistent with prior research that finds physicians are unhappy with the changes in work processes and reduced productivity caused by EHR implementation.^7,10

Patients were also unhappy; their satisfaction with almost all aspects of office visits decreased after installation of the ambulatory EHR, and, while further reductions in satisfaction occurring in subsequent stages of EHR implementation were smaller and often not statistically significant, it is striking that we find not a single positive and significant marginal effect of full two-way integration (Stage 3) in Table 3. Rather, recovery in patient satisfaction to pre-EHR installation levels, where it occurred, was driven by providers learning how to better use the new system.

The link between physician and patient dissatisfaction is consistent with research showing that provider job satisfaction influences interpersonal relationships with patients and their satisfaction with care.^33–37 This suggests that the negative reactions of both physicians and patients that we find may indicate that the dissatisfaction of providers with the new system, and its initial disruption to office operations, may have in turn increased patient dissatisfaction. In particular, our qualitative research suggests that problems initially surfaced with patient flow. For example, difficulties completing new documentation requirements sometimes slowed patient exit after appointments, or led to follow-up patient phone calls that may have caused the patient dissatisfaction we find, particularly with administrative aspects of the practice. Patients’ satisfaction may also have been negatively influenced by their interactions with dissatisfied physicians, which is consistent with prior studies that find a positive correlation between physician and patient satisfaction.^38–41 As one physician noted, “The one place you can never take the frustration is into the exam room, and it’s a challenge to do that on some days.”

Providers who brought the computer into the exam room were concerned with how it detracted from patient interactions. One physician remarked “it’s a very different doctor-patient interaction when I try to have the computer in front of me … I usually start off by saying “I apologize for the computer.” Other studies have likewise noted that EHR systems can harm physician-patient communication by inhibiting eye contact and serving as a distraction.^13,42,43 Those providers who chose instead to leave the computer outside the exam room were concerned about loss of patient interaction time because they had to review records before seeing the patient and then complete additional documentation after the visit. Prior research has also shown that healthcare workers’ satisfaction with the time that they have to accomplish their assigned tasks has the greatest effect upon patient satisfaction.⁴⁴

Complete integration of the ambulatory EHR with the inpatient EHR and greater provider experience with the system did not, in itself, reverse the negative impact on patient satisfaction, possibly because the majority of patients (GYN) did not benefit from the integration with the hospital, but only perceived the disruptions and changes in work processes that caused physician dissatisfaction. The more negative reaction of the subsample of obstetrics patients, who were most likely to benefit from access to shared information in multiple sites, however, was confined to practice administration. Yet although the satisfaction of these patients with their care providers recovered a little more quickly than did that of the entire sample, they remained more dissatisfied with other aspects of their care experience.

These findings may seem paradoxical given that prior studies of this EHR implementation found that full EHR integration, and the transmission of specific clinical data across care settings, reduced adverse birth outcomes.^16,22 One explanation is that the benefits of integration mainly apply to birth events, whereas in this study we measure patient satisfaction with outpatient OB/GYN visits. The disconnect between improvements in objective measures of birth outcomes and lower patient satisfaction with outpatient care is also related to the notion that specific practices in healthcare can have positive effects on one outcome category while, at the same time, having negative effects on other outcomes.⁴⁵

Our study has limitations that should be noted. First, we survey OB/GYN providers and patients at one healthcare system, and our results may not be generalizable to other EHR implementations. Second, we could not account for learning effects in our provider satisfaction models given that we only surveyed providers four times. Nonetheless, the average responses to questions #9 - #15 by physicians were less favorable following Stage 3 than during Stage 2, which suggests that learning did little to improve physician satisfaction. Finally, our analysis of patient satisfaction draws on surveys with a low response rate.

Despite these limitations, our results clearly indicate that the negative experience of providers with EHRs, and the challenges of incorporating EHR technology into patient care, can adversely impact patient satisfaction with the care they receive. One way to limit this negative impact on patients would be to train non-clinical staff in strategies to improve communication with patients about administrative matters during EHR transitions. Further, our results reinforce calls made by others for more thought and training on integrating computers with face-to-face interactions.^42,43,13 However, the fact that the benefits of integration were not enough to compensate for the negative impacts of the EHR on physicians suggests that healthcare systems may need to better protect physicians from the negative consequences of EHR adoption. This may include budgetary adjustments for EHR-related productivity shocks, and additional training and help in maintaining good verbal and non-verbal communication with patients while concurrently fulfilling EHR documentation requirements.^7,16 Despite the difficulty physicians had adjusting to the EHR system, they did acknowledge the resulting improvement in information transmission and its potential to improve patient care. Consequently, the impact of the EHR on both provider and patient satisfaction over a longer time period than considered in this study could be more positive.

Funding

This work was supported by Agency for Healthcare Research and Quality grant PARA-08-270 and by a Lehigh University Faculty Innovation Grant.

Contributors

CM, SS, MD, SC, DL, and MS made significant contributions to the conception and design of the study and collected the data. CM, MD, SC, XG, and JC conducted the quantitative analysis and SS conducted the qualitative analysis. All authors contributed to either the drafting or revision of the article and approved the article for publication.

Competing interests

None.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.