Using EHR audit trail logs to analyze clinical workflow: A case study from community-based ambulatory clinics

Abstract

To develop a workflow-supported clinical documentation system, it is a critical first step to understand clinical workflow. While Time and Motion studies has been regarded as the gold standard of workflow analysis, this method can be resource consuming and its data may be biased due to the cognitive limitation of human observers. In this study, we aimed to evaluate the feasibility and validity of using EHR audit trail logs to analyze clinical workflow. Specifically, we compared three known workflow changes from our previous study with the corresponding EHR audit trail logs of the study participants. The results showed that EHR audit trail logs can be a valid source for clinical workflow analysis, and can provide an objective view of clinicians’ behaviors, multi-dimensional comparisons, and a highly extensible analysis framework.

Introduction

In recent years, clinical data capture and documentation has been increasingly supported by computer systems, especially Electronic Health Record (EHR) systems, as incentivized and regulated by the HITECH Act and the Meaningful Use^1,2. While computer-based documentation (CBD) may be preferred due to its potential improvements in information availability and sharing as well as enhancement in patient care^3,4, CBD may introduce negative effects if not designed properly. One significant negative effect is that a poorly designed CBD system contributes to cumbersome documentation activities and inaccurate and incomplete data capture when it fails to consider clinical workflow⁵.

In order to address this issue and design a workflow-supported CBD system, a critical first step is to understand and measure the current workflow and documentation behaviors, which has been frequently achieved by gold standard techniques such as Time and Motion (T&M) studies⁶. T&M studies, however, are often costly and incapable of collecting data in a large scale because of the challenges in recruiting numerous participants and observers as well as coordinating observation sessions. Meanwhile, T&M studies are inherently limited in accuracy due to human observers’ cognitive overload and biases, resulting in suboptimal quality of behavioral data such as incompleteness and misinterpretation.

An alternative method to make sense of clinical workflow is through the secondary use of EHR data, which has drawn significant attention from researchers. For example, Ozkaynak et al. (2015) characterize the workflow in pediatric emergency rooms by analyzing encounter and diagnosis data using Markov chain and visualization tools⁷. Kirkendall et al. (2014) understood the workflow of medication order-dosing alerts by simulating order activities in a testing EHR environment⁸. Redd et al. (2014) measured the efficiency of documentation by extracting the chart open and completion time in each patient encounter⁹. These studies, although suitable for their purposes, only utilized EHR data on a surface level. The data and methods in these studies did not have the ability to uncover more nuanced behavioral patterns of a large number of individual clinicians.

The clinicians’ behavioral patterns can be uncovered by a special type of EHR data, EHR audit trail logs, which automatically and minutely record clinicians’ EHR interactions for security and privacy purposes. These logs can be a cost-effective alternative to characterize clinicians’ documentation behaviors and have the potential to facilitate the understanding of clinical workflow and processes. For example, Hirsh et al. used audit files to understand time spending in primary care¹⁰. However, to our knowledge, there is no study evaluating EHR audit trail logs as a valid source for clinical workflow analysis. This study therefore bridges this methodological gap by assessing the ability of these logs to provide supportive evidence to known workflow changes. Specifically, the study analyzed a sample of EHR audit trail log data extracted from an ambulatory healthcare organization. This organization launched an EHR enhancement project in mid 2013, resulting in subsequent workflow and behavioral changes as identified in our previous study¹¹. The present study chooses three findings to evaluate the ability of the EHR audit trail logs to provide supportive evidence. The implications of utilizing such log data for clinical workflow analyses are discussed. It is worth noting that these evaluation standards were validated based on the triangulation of our previous semi-structured interviews, ethnographic observations, and T&M studies. The qualitative analysis is out of the scope of this paper and will be published in another venue.

Methods

Study Site

The participating study site was a not-for-profit ambulatory healthcare organization located in an urban area in the western United States (Organization West). This organization contained more than 20 branches and affiliated locations including both primary care and specialty clinics. Organization West served several states over a widely rural area, with over 50% of patients being rural residents and around 50% of who are on Medicare/Medicaid. After nearly 10 years of use of a vender EHR system, Organization West launched a significant update to enhance the system capability in 2013, including an electronic patient homepage, a standardized message center, a new computerized provider order entry (CPOE), and an e-prescribing functionality. During the EHR enhancement, one of the clinics experimented with a “Core Team Model”, in which a provider was teamed up with three clinical staff members to eliminate inefficiencies and patient wait times.

Data Collection

This study targeted three clinics in Organization West, including a primary care clinic in the main hospital (Primary Care 1), another primary clinic that experimented with the “Core Team Model” (Primary Care 2), and an Ear Nose Throat clinic (Specialty Care). A total of 24 clinicians were participated and equally distributed in each clinic. For each primary care clinic, the study participants included four providers, two nurses, one medical assistant, and one receptionist. For the specialty care clinic, the study participants included five providers, two nurses, one medical assistant, and no receptionist. The providers included in the study were physicians (MD), ophthalmologists (OD), nurse practitioners (NP), and physician assistants (PA). The participants’ EHR audit trail logs were extracted before, during, and after the EHR system changes. This data allowed the comparison of clinicians’ behavioral changes in many dimensions, e.g. primary care vs. specialty care; with and without the “Core Team Model”; and providers vs. non-providers. Since the quantitative analysis team (DW, NS, and KZ) was not physically located in the study sites, the log data were de-identified and exported as CSV files for analysis.

Data Manipulation

The log data were stored in a standalone SQLite database and manipulated using Structured Query Language. Specifically, the log data were categorized into three stages based on the timestamp: 1) PRE: log records comprised of a period of eight weeks prior to the launch date, 2) DURING: log records for a period of four weeks after the launch date, and 3) POST: log records of the eight weeks after the end of the DURING stage. The analysis in this study only focused on the data in the PRE and POST stages to examine the potential differences. The start and end of a week was set from Monday to Sunday. Moreover, the date and time information of the timestamps were parsed and stored separately, e.g. date, hour, and day of the week. The time information was further categorized into three types: 1) Regular hours (REG): time between 7am and 6pm from Monday to Friday; 2) off hours (OFF): time not in regular hours from Monday to Friday; and 3) weekend hours (WKN): anytime during Saturday and Sunday only. The start and end of the regular hours was determined based on our previous interviews and observations.

In addition, the clinical roles in the log data were marked as Provider (P), Nurse (N), Medical Assistant (A), and Receptionist (R). Note that the study created seven new event categories in addition to the original event names and tasks defined by the EHR vendor, including: 1) Communication, 2) Entering, 3) Login/Logout, 4) Processing, 5) Reading, 6) Printing, and 7) Other. Each event was first mapped to a category separately by the first two authors. Then, the mappings were discussed and an agreement was reached. If a consensus of an event mapping cannot be reached, this event will be categorized as “Other”.

Hypotheses

The primary objective of this study was to evaluate the ability of EHR audit trail log data to provide supportive evidence to the known workflow changes. In particular, it aimed to reflect documentation-related behavioral changes among clinicians after the rollout of an EHR enhancement project. Three previously validated findings were chosen as the reference standards of the present study. The first evaluation involved a workaround order-placing behavior in Primary Care 1, where a provider likely had to prescribe a medication more than once in order to achieve the desired dose in the POST stage. For example, a provider would place two medication orders, one at 1.5 mg and the other at 0.5 mg, to equal the total intended dose of 2 mg. Since the order dosage was not captured in the log data, the providers in Primary Care 1 were alternatively hypothesized to have more number of order logs daily. Their patient volume in the PRE and POST stage was hypothesized to remain stable.

H1: After the EHR enhancement, the providers in Primary Care 1 had more number of order logs, but their patient volume remained stable.

The second evaluation was a shift of tasks in Primary Care 2, where providers spent less time on computer-related tasks after the EHR enhancement, while other clinicians consumed those tasks due to the implementation of the Core Team model. The providers in Primary Care 2 were hypothesized to have fewer “Computer Entering”-related logs in the POST stage, compared to the PRE stage. On the other hand, non-providers in the POST stage were hypothesized to have more “Computer Entering”-related logs.

H2: After the EHR enhancement, the providers in Primary Care 2 had fewer number of “Computer Entering”- related logs, while the non-providers had more such logs.

The third evaluation was the phenomenon of “deferred documentation” in Specialty Care clinic, highlighting differences in dictation. It is noted that providers in the PRE stage would dictate findings between patient visits, while in the POST stage they tended to document patient visits at the end of a day. It is hypothesized that providers in Specialty Care would have an increased number of logs in the off hours (and even weekends) in the POST stage.

H3: After the EHR enhancement, the providers in Specialty Care had more number of logs in the off hours.

Data Analysis

The data were analyzed through self-developed Python scripts, which were implemented using libraries such as pandas, numpy, scipy, and xlrd. The data analysis used two primary measures, namely, the number of logs and the percentage of logs. The analysis began with a statistical summary of all logs, followed by the statistical tests of the three hypotheses. For the first hypothesis (H1), the log records related to “adding an order” were selected based on keywords. The total number of orders placed per day was counted using the order identifiers stored in the logs. The days without any orders placed were dropped. On the other hand, the daily patient volume was estimated by the total number of patients accessed per day. For the second hypothesis (H2), the log records were manually coded into the seven high level categories, followed by the calculation of the number and percentage of logs in each category (e.g. Computer Entering). For the third hypothesis (H3), the same measures as H2 were used to quantify the off-hour and weekend logs. All the above measures in the PRE and POST stage were compared using a two-tailed, unequal variance t-test.

In addition, in H2 the fragmentation of the log events was assessed based on their categories. This measure calculated a fragmentation score representing the level of cognitive overload (12). The higher the score is, the more switches between categories the clinician had in the log data, and the more likely this person was suffered from cognitive overload. It is worth noting that a person may have multiple logs with varying categories but identical timestamp because multiple documentation behaviors may happen, or be triggered, at the same time. Fragmentation was detected if there was no overlap between any event categories of the current timestamp and its previous one. A daily fragmentation score was defined as the total number of detected fragmentation over the total number of timestamps on that day. The clinicians’ daily fragmentation scores in the PRE and POST stage were compared using the same t-test technique as the hypotheses.

Results

Statistical Summary

A total of 2.65 million EHR audit trail logs were extracted from Organization West in both PRE and POST stage, for a period of eight weeks each. Overall, the log volume of Primary Care 1 & 2 was approximately the same, whereas the log volume of Specialty Care was half of the previous two (Table 3.1). This is likely due to the differences of clinical practices between a primary and a specialty care clinic. In terms of the log volume by clinical roles, the providers in Primary Care 2 had a much lower number than their nurses (87,600 vs. 141,090). In comparison, the average number of logs of the providers and nurses in Primary Care 1 were more comparable (111,351 vs. 116,078).

Table 3.1.

Summary of the sample EHR audit trail log data.

Clinic	PRE (8 wks)	Launch Date	POST (8 wks)	Total (# log)	Provider (avg. total)	Nurse (avg. # log)	Assistant (avg. # log)	Receptionist (avg. # log)
Primary Care 1	5/20/13 7/14/13	7/16/13	8/12/13 10/6/13	1,149,295	4 (111,351)	2 (116,078)	1 (215,347)	1 (256,386)
Primary Care 2	6/24/13 8/18/13	8/20/13	9/16/13 11/10/13	971,210	4 (87,600)	2 (141,090)	1 (143,758)	1 (194,868)
Specialty Care	8/5/13 9/29/13	10/1/13	10/28/13 12/22/13	529,999	5 (64,099)	2 (74,045)	1 (61,410)	-

A further examination of the log volume considering the PRE and POST stage is listed in Table 3.2. This table shows that the nurses in Primary Care 1 had a 20% decrease of their log volumes and the providers had a 20% increase and in the POST stage. On the other hand, the nurses in Primary Care 2 had a 11% increase in their log volumes while the providers maintain a similar number of logs. This observation might be attributed to the implantation of the Core Team model in Primary Care 2, which increased the nurses’ documentation responsibility.

Table 3.2.

The average number of logs in the PRE and POST stage (providers and nurses).

Clinic/Role-Stage	Provider (4)			Nurse (4)
	PRE	POST	DIFF.	PRE	POST	DIFF.
Primary Care 1	50,591	60,759	+20.1%	64,664	51,414	-20.5%
Primary Care 2	43,452	44,148	+1.6%	66,782	74,308	+11.3%
Specialty Care	36,605	27,493	-24.9%	46,680	27,365	-41.4%

Hypothesis One

As shown in Table 3.3, the statistical test indicated that the providers in Primary Care 1 placed significantly more orders in the POST stage (all p<0.05) and maintained the same number of patients seen, which confirms our hypotheses. The same statistical test was further extended to the providers in Primary Care 2, which didn’t show the same pattern. As shown in the second half of Table 3.3, two of the four providers in Primary Care 2 had no significant change in their order numbers, with one of them seeing more patients in the POST stage. This finding suggests that changes of EHR systems may impact clinicians’ behaviors differently even though they operate within the same heath system and have similar clinical practice.

Table 3.3.

Number of orders placed and patients seen for each provider in the two primary care clinics.

Clinic	Measure	Provider	PRE	POST	P-value
Primary Care 1	Number of Orders Placed	BC1-04	12.00	35.25	0.0 *
		BC1-05	42.35	53.15	0.0192 *
		BC1-09	44.68	53.53	0.0433 *
		BC1-10	15.94	30.14	0.0 *
	Number of Patients Accessed	BC1-04	42.50	34.75	0.3491
		BC1-05	42.57	42.58	0.9971
		BC1-09	31.45	35.32	0.1939
		BC1-10	40.47	41.94	0.6158
Primary Care 2	Number of Orders Placed	BC2-02	23.04	32.11	0.0066 *
		BC2-03	16.69	16.07	0.8386
		BC2-04	28.89	38.12	0.0342 *
		BC2-16	27.38	33.84	0.1826
	Number of Patients Accessed	BC2-02	31.77	28.34	0.1475
		BC2-03	29.47	35.11	0.0399 *
		BC2-04	36.39	34.46	0.5024
		BC2-16	23.14	22.04	0.6417

* p-value < 0.05

Hypothesis Two

The second evaluation began at coding the log events to the seven high-level categories. The first two authors achieved 81% agreement on coding the event categories initially, and resolved most of the discrepancies afterwards. Table 3.4 lists top ten frequent events that were assigned to “Computer Entering”, and their vender defined names and types as well as their frequencies in Primary Care 2.

Table 3.4.

Top 10 “Computer Entering” events

EVENT TYPE	EVENT NAME	COUNT	PCT
Maintain Person	Chart Access Log	141,616	42%
Maintain Person	Ensure	70,647	21%
Maintain Encounter	Patient-Provider Relations	22,068	7%
View Orders	Modify Details	16,719	5%
Maintain Encounter	Ensure	14,681	4%
Maintain Person	Patient	12,921	4%
Maintain Clinical Events	Write/Update Results	11,331	3%
Maintain Order	Add	9,770	3%
Maintain Order	Tasks	5,939	2%
Maintain Encounter	Prsnl Relationship	4,894	1%

The statistical test examined the differences of the “Computer Entering”-related events between the PRE and POST stage. The results (Table 3.5) showed that the participated nurses (BC2-11 & 19) and the medical assistant (BC2-22) had a significant increase (3-5%) in their percentage of computer entering logs as hypothesized. However, the receptionist (BC2-45) had an unexpected drop in their percentage of entering-related logs (53% – 49%). On the other hand, the providers had a slight but not significant decrease in their number and percentage of logs, which is also unexpected.

Table 3.5.

The average number and percentage of “Computer Entering” logs in the PRE and POST stage.

Clinical Role	Participant ID	Number of Logs			Percentage of Logs
		PRE	POST	p-value	PRE	POST	p-value
P	BC2-02	364.19	354.72	0.7983	24%	24%	0.6771
P	BC2-03	484.57	486.17	0.9743	29%	27%	0.0889
P	BC2-04	327.75	294.79	0.4302	22%	21%	0.4339
P	BC2-16	341.48	318.71	0.6085	27%	27%	0.9254
N	BC2-11	515.22	655.14	0.005*	27%	32%	0.0*
N	BC2-19	515.33	603.91	0.0516	24%	27%	0.0*
A	BC2-22	673.34	683.12	0.8806	28%	31%	0.0*
R	BC2-45	1375.36	1236.83	0.0737	53%	49%	0.0*

^* 0.05 significance

The workflow fragmentation analysis of these providers (Table 3.6) indicated that three out of four providers had a lower level of fragmentation, although only one of them was statistically significant. The other providers had a slight but not significant increase in the level of fragmentation (0.398 – 0.401). Together, the analysis of the second evaluation supports the hypotheses of shift clinical workflow, where the nurses and the medical assistant had an increased percentage of “Computer Entry”-related logs. While the providers didn’t have a significant increase, they had an improved level of fragmentation in the POST stage, which may help reduce their cognitive overload, leading to their positive attitude toward the EHR change (i.e. more time with patients).

Table 3.6.

The workflow fragmentation analysis of the providers in the Primary Care 2. The higher the score is, the more switches between log event categories the clinician had, the more likely cognitive overload happened.

Participant ID	PRE	POST	Difference	p-value
BC2-02	0.428	0.361	-0.067	0.0043*
BC2-03	0.388	0.365	-0.023	0.2669
BC2-04	0.358	0.306	-0.052	0.1305
BC2-16	0.398	0.401	0.003	0.8725

* 0.05 significance

Hypothesis Three

The initial statistical test of the providers’ weekly off-hour logs in Specialty Care, unfortunately, did not find any significant change at the 0.05 level. The statistical tests of the weekend logs also found no significant differences. A further exploratory analysis using visualization (line charts) suggested that the providers in Specialty Care had an increased number of logs right after 5pm (Figure 3.1). The data were therefore re-grouped and tested on an hourly basis. As shown in Table 3.8, four of five providers in Specialty Care had a significantly higher number of logs. They deferred documentation activities in at least one of the hours between 5pm and 10pm. The other provider (BC3-01) seemly had already deferred the documentation activities in the PRE stage, which makes the EHR change less impactful in the POST stage.

Figure 3.1.

The visualization of the number of off-hour logs of the provider BC3-04. X-axis represents the hours of a day and Y-axis represents the number of logs of all providers.

Discussion

This case study demonstrated the validity of using EHR audit trail logs to analyze clinical workflow. The results indicated that EHR audit trail logs contained rich information of clinicians’ documentation behaviors. The analyses base on the log data successfully produced supportive evidence to the three known behavioral and workflow changes. For the first hypothesis, the data confirmed that the providers in Primary Care 1 placed more orders and maintained the same patient volume. For the second hypothesis, the data supported that the nurses in Primary Care 2 had more “Computer Entering”-related activities, while the providers may have a lower cognitive overload. For the third hypothesis, the data indicated that the providers in Specialty Care deferred their documentation behaviors, although each provider had a different working timeframe.

The study showed several benefits of using EHR audit trail logs to analyze clinical workflow. First if all, since the log data were recorded automatically and consistently, an analysis of one clinic can be easily applied to another clinic with almost no extra cost. Take the first hypothesis for example, the changes of the order numbers and patient volumes was initially examined in Primary Care 1 and then directly applied to Primary Care 2 with nearly no extra effort. This benefit of easy extension makes this method more attractive than other workflow analysis tools, e.g. Time and Motion studies and work sampling.

Another benefit of using EHR logs for workflow analysis is to provide an objective view of clinicians’ EHR behaviors and to verify the findings derived from other methods. Take the second hypothesis for example, the previous study suggested that the providers in Primary Care 2 had more time for patient communication because they shifted documentation responsibilities to other clinicians as a result of the Core Team model. The analysis of the present study, however, showed that the providers’ EHR log volume did not drop significantly although other clinicians, especially nurses, indeed had a significant increase of the log volume. One possible explanation is that EHR audit trail logs may not fully capture the documentation process as they are not design for such intension. Another explanation of this phenomenon, as suggested by the workflow fragmentation analysis, is that the providers may have less fragmented documentation activities, leading to a smoother workflow and higher quality of patient interactions.

Still another benefit of using EHR logs for workflow analysis is the ability to compare and contrast results in multiple dimensions. In this study, the dimensions included clinical roles (providers vs. non-providers), care types (primary vs. specialty), care models (normal vs. the core team model), time (regular vs. off-hour), and event categories (Entering vs. Other). Moreover, EHR audit trail logs enable detailed comparison showing the differences of individual clinician’s behaviors. Take the third hypothesis for example, the analysis not merely supported the hypothesis of deferred documentation in Specialty Clinic, but further indicated which hour(s) the documentation activities were deferred to, which were varying among the five providers.

Although the rich information in EHR audit trail logs has several benefits, it can introduce analytical challenges. One such challenge is the high complexity and high level of noise in the log data. It could be very difficult to identify meaningful patterns solely based on EHR log data. The present study voided this problem by utilizing the findings from the previous study to shed light on the analysis. This suggests that using EHR logs for clinical workflow analysis may require mixed methods, e.g. integrating the qualitative results to the quantitative analysis. It may also require extensive effort on data exploratory, which can be supported by visual analytics techniques. Another challenge is that EHR logs may require high-level categorization to reduce their complexity and to make them more meaningful. For example, in the present study, the logs were manually categorized into seven groups in order to compare “Computer Entering” activities with other kinds of activities. Since each vendor has a different mechanism capturing audit trail logs, and since each study has different analysis goals, it seems that researchers would need to spend a significant amount of time labeling EHR logs. Data mining and machine learning techniques may help facilitating this process.

This study has several limitations. First, this study only analyzed EHR audit trail logs of one organization in a community-based, ambulatory setting. Since this is a proof-of-concept study, we believe that the scope is adequate. Future research can consider expanding the scope to analyze EHR logs in an inpatient setting, of different EHR systems, and/or from multiple organizations. Second, the study did not obtain the EHR audit trail logs of all clinicians in the targeted organization, but only the logs associated with the 24 participants in the three selected clinics. This limitation prevents the study from performing a comprehensive patient-centered analysis because the selected participants may not work on the same patients in the study period. Third, although we conducted interviews, observations, and T&M studies to understand workflow and behavioral changes in our previous study, we did not compare step-by-step how clinical behaviors are recorded to form audit trail logs. Rather, we understood the log data based on the description of the attributes, the sequences, and the timestamps of the logs. We formed an interdisciplinary research team to mitigate potential biases in this interpretation. Next, we only employed two volume-related measures in this study. We have been working on applying data mining techniques, especially in sequential pattern mining, to uncover unexpected but interesting workflow patterns from the EHR audit trail logs. Last but not least, many results were not significant at the 0.05 level, and even not significant at the 0.1 level. However, a statistically insignificant difference may still have clinical impact. These insignificant differences still showed changes (increase or decrease of the means) aligned with our hypotheses.

Conclusions

This study evaluated the validity of using EHR audit trail logs for clinical workflow analysis. The three evaluations successfully provided supporting evidences and justify the use of this type data. Future research includes developing a generalized framework capable of analyzing log data from any EHR systems, advanced computational and visual-based methods to uncover more nuance patterns of clinicians’ behavioral changes, and a real-time monitoring system to detect workflow bottlenecks, physician burnout, and opportunities to support medical education.

Table 3.7.

The average number of daily off-hour log in the PRE and POST stage.

Participant ID	Flour	PRE	POST	p-value
BC3-01	5pm	173.26	147.7	0.7476
	6pm	118.96	186.5	0.3326
	7pm	61.93	86.25	0.6350
	8pm	11.07	0.50	0.1782
	9pm	0	0	-
BC3-04	5pm	0.91	16.65	0.0263 *
	6pm	0.36	19.10	0.0373 *
	7pm	0.33	12.00	0.064 **
	8pm	4.61	12.05	0.2927
	9pm	3.33	0	0.4416
BC3-05	5pm	53.14	64.42	0.6525
	6pm	17.05	13.50	0.8124
	7pm	31.97	0.04	0.1661
	8pm	12.11	0.29	0.3917
	9pm	0.14	26.75	0.0393 *
BC3-10	5pm	71.74	88.26	0.6039
	6pm	43.71	19.22	0.2146
	7pm	0	0.35	0.0978 **
	8pm	5.03	0.35	0.4277
	9pm	0	0.13	0.1174
BC3-14	5pm	0.72	6.78	0.0626 **
	6pm	0	2.00	0.1717
	7pm	0	0.09	0.2417
	8pm	0	0	-
	9pm	0	0	-

* 0.05 significance;

** 0.10 significance.