Secondary Use of Electronic Health Record Data for Prediction of Outpatient Visit Length in Ophthalmology Clinics

Abstract

Electronic health record systems have dramatically transformed the process of medical care, but one challenge has been increased time requirements for physicians. In this study, we address this challenge by developing and validating analytic models for predicting patient encounter length based on secondary EHR data. Key findings from this study are: (1) Secondary use of EHR data may be captured to predict provider interaction time with patients; (2) Modeling results using secondary data may provide more accurate predictions of provider interaction time than an expert provide; (3) These findings suggest that secondary use of EHR data may be used to develop effective customized scheduling methods to improve clinical efficiency. In the future, this has the potential to contribute toward methods for improved clinical scheduling and efficiency.

Introduction

Electronic health records (EHRs) system promises a number of benefits, including improvement of the quality, efficiency, and cost.^1-5 Rapid EHR adoption has occurred in the United States.⁶ This transition from paper-based medical records to EHRs has been supported by federal initiatives,^7,8 and is quickly transforming the process of health care delivery.⁹ However, there are persistent concerns about negative impacts of EHR implementation on clinical productivity.^10-12 For example, our previous studies showed that ophthalmologists at Oregon Health & Science University (OHSU) saw 3-5% fewer patients and spent over 40% additional time for each patient encounter after EHR implementation in 2006.¹²

Efficiently scheduling clinic appointments to improve patient waiting time provider-patient interaction time is a challenging task. In some specialties, the workflow is much more complicated. For example, during a half-day clinic session, ophthalmologists typically see 15-40 patients or more. To efficiently see this number of patients, high-volume clinical providers typically utilize the multiple exam rooms simultaneously, examine patients at different stages of their workup, and integrate ancillary staff and trainees into the clinical workflow.⁹ Thus, it is difficult to study and test appointment scheduling strategies in clinical settings. Few studies have examined clinic visit times or analyzed the time spent in each portion of a patient visit across different specialties.^13-15 The interaction time between the healthcare providers and the patients is one of the most important factors associated to patient satisfaction in clinics.^16-18 Also, patient wait time and satisfaction were the sources of time pressure on the healthcare providers.^19,20 Improving clinical workflow requires being able to predict which patients will be of lower, average, or greater complexity, but there is no easy way to do this.

Secondary use of EHR data creates an opportunity to address this gap in knowledge by providing information which may help improve workflow.²¹ We have previously shown that EHR timestamp data may be used to develop simulation models which result in patient scheduling strategies that improve patient wait time and clinic length.^22,23,24 These simulation models rely on categorizing patients into predicted complexity (i.e. length of time required for patient-provider interaction). From this perspective, an accurate and reliable way to predict patient complexity would have significant benefits for clinical workflow modeling.

The purpose of this paper is to address this gap in knowledge by developing analytical models to predict provider-patient interaction time using secondary EHR data. We will examine which clinical and demographic features that are most important for these analytical models. These models are validated by comparison with actual provider-patient interaction times, and by comparison with predictions by an experienced physician. This study was performed in an academic ophthalmology department because it is a high volume, fast-paced specialty where clinical efficiency and effective scheduling are paramount.

Methods

The study was approved by the Institutional Review Board at Oregon Health and Science University (OHSU).

Study environment

The study was conducted in the ophthalmology department at OHSU, which is a large academic medical center in Portland, Oregon. The ophthalmology department includes over 50 faculty providers, who perform over 130,000 annual outpatient examinations. The department provides primary eye care, and serves as a major referral center in Pacific Northwest and nationally. An institution-wide EHR system (EpicCare; Epic Systems, Madison, WI) was adopted in 2006; all ophthalmologists at OHSU use Epic for clinical care and billing activities.

Data collection

This study used data from office visits from January 1, 2015, to December 31, 2016, with 5 faculty providers within the pediatric ophthalmology division at OHSU. Time-stamps and related data about the office visits were abstracted from the enterprise-wide clinical warehouse at OHSU. Additionally, audit log timestamp data was used to calculate the provider interaction time with the patient, which served as the primary outcome variable for our study.

Office visits were excluded if provider interaction time was less than 1 minute or greater than 120 minutes. This was done to minimize the effects of outliers on our analysis. A total of 8,675 office visits met our inclusion criteria and 2,455 (22%) office visits were excluded.

Analytical Models

Overall, we developed multiple linear regression and random forest regression²⁴ models to predict the patient-provider interaction time using time-related variables and the characteristics of each encounter. Details of these models are described below.

Outcome Variable: Patient-Provider Interaction Time

Patient-provider interaction time was calculated based on methods that we have previously described.²³ In this study, provider interaction time represents the total interaction time of all providers and staff (faculty providers, fellows, residents, and technicians) who participated in the appointment. Using this variable instead of the total appointment length is potentially simpler since it does not include patient wait time. In one of the random forest models, provider interaction time was transformed into a categorical variable “complexity” with values of “long “, “medium”, or “short”. “Short” appointments were in the bottom 20% (< 12 minutes) of provider interaction time, while “long” appointments were in the top 20% (> 29 minutes).

Predictor Variables in Analytic Models: Time-Related

There are twelve predictor variables in this study included two time-related variables and ten variables related to other clinical and demographic features. Average provider interaction time is a continuous variable that captures the average provider interaction time of a patient’s previous visits, excluding the first, as a new patient appointment is known to be longer than average and therefore would not accurately reflect a future visit’s time. The rationale for this variable was that previous encounter length may be predictive of future encounter length. The Arrival Interval is the duration between patient check-in time and the scheduled visit time. This variable can either be positive (late arrival) or negative (early arrival). The rationale for this variable was that arrival time may affect workflow and be predictive of future encounter length.

Predictor Variables in Analytic Models: Other

An ICD-9 diagnosis code for each office visit; our dataset included a total of 1097 distinct ICD-9 diagnosis codes. To simplify and allow more precise parameter estimates, we grouped the code into 21 categories base on the frequency of the diagnosis. The 20 most common ICD-9 diagnosis codes were the first 20 categories and all other codes were categorized as “Other”. The rationale for this variable was that diagnosis may affect patient complexity and be predictive of future encounter length.

Age was grouped as 6 categories: infant (0-1 years), toddler (1 – 3 years), pre-school (3 – 6 years), school age (6-13 years), adolescent (12-18 years), and adult (<=18 years). The rationale for this variable was that patient age may affect ease of the patient-provider interaction and therefore be predictive of future encounter length.

Reason for visit was a categorical variable with the following categories: “follow-up”, “post operative follow-up”, “pre-operative evaluation”, and “others”. The rationale for this variable was that reason for visit may affect complexity and therefore be predictive of future encounter length.

Patient financial class was categorized as “self-pay”, “Medicaid”, “Medicare”, “health insurance contracted with OHSU”, “health insurance non-contracted with OHSU”, and “others”. The rationale for this variable was that health insurance status may affect patient complexity and be predictive of future encounter length.

Patient’s native language was categorized as “English”, “non-English with interpreter”, and “non-English without interpreter.” The rationale for this variable was language may affect ease of the patient-provider interaction and therefore be predictive of future encounter length.

Clinic hour was the hour of the patient’s check in time relative to the scheduled half-day clinic session. Other variables included the number of trainees in the session, the half-day clinic volume, whether the patient was dilated, and which provider the patient was seeing. The rationale for these variables were that they may affect the nature of the patient- provider interaction and therefore be predictive of future encounter length.

Models

Data processing and analysis was conducted using R programming language version 3.4.3.²⁵ We built three models that predict the provider interaction time (outcome variable) using our predictor variables: 1) a multiple linear regression, 2) a random forest regression,^24,27 and 3) a random forest classification. The first two models predicted the outcome variable as a continuous time variable and the third model predicted the outcome variable as a categorical one: short, medium, or long interaction times. The model constructs multiple decision trees at training time. Each of these trees is constructed from a random subset of the total predictor variables in a given dataset. An estimate of the classification error-rate can be used to predict out-of-the bag (OOB) sample and gets the average of the out-of-the bag predictions to grow the forest.^26,27 The predictions could be quite accurate if the forest is big enough. Random forest classification was used to predict the categorical provider interaction time. Each encounter by a patient is modeled as a function of the predictor variables described above. All data were split into the training data set and the testing data set as 75% and 25%.

Results

Descriptive data.

Characteristics for the patient encounters are shown in Table 1. A total of 8,675 patients encounters met inclusion criteria. The average provider interaction time was 21 minutes. On average, patients arrived 9 minutes prior to their scheduled time. Most patients were school age or pre-school age. The three most common diagnoses were strabismus, amblyopia, and intermittent heterotropia.

Table 1:

Descriptive characteristics

Regression models results

Results from multiple linear regression and random forest regression using the continuous outcome variable are displayed in Table 2. We presented the five most important features of random forest regression and five correlated predictor variables. Important features indicate that the features are more closely related with outcome variable and contribute more for variation of the outcome variable.²⁷ The statistically significant variables associated with the provider interaction time include the ICD9 - Astigmatism, faculty provider, reason of visit (Post Operative visit), age (baby), financial class (Medicare), pupil dilation, average prior provider interaction time, the number of trainees in the session, and the clinic hour. The R-squared of the multiple linear regression was 0.21. In other words, the model explains approximately 21 percent of the variability of the provider interaction time. 20 percent of the variability of the provider interaction time can be explained by the random forest regression. The mean squared error (MSE) of the multiple linear regression was 83.63 and the MSE of the random forest regression was 95.38.

Table 2.

Top 5 Predictors of provider interaction time in regression models. The average prior interaction time is the biggest predictor in both models.

Random forest regression		Multiple linear regression
Variable	%IncMSE	Variable	Coefficient	P	SE
Faculty provider	50.27	Average prior interaction time	0.179	<0.001**	0.020
Pupil dilation	48.08	Number of trainees in the session	0.638	<0.001**	0.060
Average prior interaction time	25.06	Clinic hour	-1.487	<0.01*	0.582
Age	19.45	Dilation	5.67	<0.001**	0.273
ICD9 -diagnoses	17.46	Faculty provider	-2.76	<0.001**	0.832

%IncMSE: How many percent increasing of the mean squared error of predictions

^*Significant value.

We created visualization to compare the distribution of the actual provider interaction time and predicted provider interaction time in Figure 1. In addition, the difference of the provider interaction time between predicted value and actual value was shown in Figure 2.

Figure 1:

Distribution of prediction result and actual time for regression models. Both models’ distributions of prediction values are close to the actual values for interaction time.

Figure 2:

Difference between predicted and actual values for regression models. The distribution of differences for both models are similar.

Classification models results

The results from the random forest classification using the categorical outcome variable are presented in Table 3. The five most important features of random forest classification variables were shown. The five most important features included: pupil dilation, average prior provider interaction time, faculty provider, ,age, and ICD-9 diagnosis code. Comparing the predicted complexity with actual complexity employed the accuracy of the model. In addition, we validated the prediction results by comparing them to a faculty provider prediction data. Table 4 presents the random forest classification, expert provider prediction result, and actual result. Random forest classification had an accuracy score of 0.65, while provider prediction had an accuracy score of 0.41. The difference of important features between random forest models is based on the ranking method. In the regression model, %IncMSE was used to rank the variables, which means how many percent increasing of the mean squared error of predictions. In the classification model, the ranking method is mean decrease accuracy (MDA), which represents how much removing each variable reduces the accuracy of the model.^26,27

Table 3.

Top 5 predictors of provider interaction time in random forest classification model. The average prior interaction time is the top predictor.

Random forest Classification
Variable	MDA
Faculty provider	50.47
Dilation	49.14
Average prior interaction time	21.03
Age	18.22
ICD9-diagnoses	17.94

MDA: Mean decrease accuracy

Table 4:

Classification Comparison of random forest model and provider prediction. The random forest model classifies the provider interaction time more accurately than the provider

Discussion

This study evaluates the use of analytic models based on secondary EHR data to predict patient exam length. There are three key findings from this study: (1) Secondary use of EHR data may be captured to predict provider interaction time with patients; (2) Modeling results using secondary data may provide more accurately predict provider interaction time than an expert provider in this study (3) These findings suggest that secondary use of EHR data may be used to develop effective customized scheduling methods to improve clinical efficiency.

The first key finding is that secondary use of EHR data may be captured to predict provider interaction time with patients. Generally, the purposes of using secondary EHR data include clinical research, quality assurance, public health, administrative, and billing.^21,28,29 In this study, we extend these applications by demonstrating that secondary EHR data may be used for operational purposes, and that data needed to predict provider interaction time could be mined from the EHR. Predictions from these models were highly accurate compared to actual measured times (Figure 1 and Figure 2).

The most important predictor variable identified in this study was the historical averaged provider interaction time, faculty provider, and dilation. They were highly correlated with the provider interaction time in all models (Table 2). On the other hand, “clinic hour” showed negative correlation with the provider interaction time, suggesting that the provider-patient interaction time decreases when patients are scheduled late in a session. This might be explained because, toward the end of a clinic sessions, providers may feel more time pressure and decrease the time spent interacting with patients. Comparing the significant features of the linear regression and random forest models, we found that the age was significant in the random forest model, but not in the multiple linear regression model.

Furthermore, the number of trainees in the session showed positive correlation with the provider interaction time in multiple linear regression. This is consistent with our previous study that showed the presence of trainees was associated with longer appointment times.³⁰ Prior studies have shown that lower income patients may receive less time with their clinicians,³¹ but we did not find financial class to be significantly correlated with provider interaction time in this study.

The second key finding is that modeling results using secondary data may provide more accurately predict provider interaction time than an expert provider in this study. Data analysis in this study showed that the random forest classification model using the categorical outcome variable provided an accurate prediction (accuracy 0.65) (Table 4). Moreover, both random forest regression and multiple linear regression models using the continuous outcome variables explained approximately 20% of the variability of the patient-provider interaction time. On the other hand, the accuracy of provider interaction time prediction by an experienced physician is 0.42 (Table 4). The physician judged the interaction time based on her expertise and clinical experience; several factors played important roles, such as patients’ diagnosis, patients’ previous interaction time, and the time requirements of the appointment. The physician’s prediction result does not appear to be very accurate. Because this study used a single physician expert, the prediction result needs further investigation with multiple expert physicians. Overall, these results suggest that analytic models using secondary EHR data can provide accurate predict provider interaction time, and the accuracy of prediction may be better than an experienced clinician. This has important implications for the possibility of future electronic systems that perform real-time patient scheduling using analytic data methods.

The third key finding is that secondary use of EHR data may be used to develop effective customized scheduling methods to improve clinical efficiency. We have previously shown that simulation models may be used to develop improved patient scheduling templates which decrease patient wait time and clinic length.^22-24 The templates in those studies were based on predictions of patient wait time and complexity by an experienced clinician. In the future, these predictions might be incorporated into electronic systems that perform real-time patient scheduling using data analytic methods. This has potential to improve patient wait time, patient satisfaction, and clinician satisfaction.

There are several limitations in our study. First, the EHR timestamps do not always capture the interaction time with the patients when providers are not using the EHR. That said, we have previously validated that these timestamps were accurate across a wide range of ophthalmology providers at a single institution.^22-24 Second, our study was limited to 5 pediatric ophthalmologists and one expert reviewer at single institution. The extent to which these findings generalize across different medical specialists at different institutions is not known, and additional studies will be required. Third, only three analytic models were tested in our study, and additional studies may be warranted. Finally, we consulted only one expert; more experts will be needed in future study.

Conclusion

One challenge of EHR implementation in ophthalmology has been increased time requirements by physicians.^34-35 In this study, we show that secondary use of EHR data can be applied to support analytic models to predict patient encounter length. These models may provide more accurate prediction results than an expert provider in this study, and have potential to contribute toward improved clinical scheduling and efficiency in the future.