Context is Key: Using the Audit Log to Capture Contextual Factors Affecting Stroke Care Processes.

Abstract

High quality patient care through timely, precise and efficacious management depends not only on the clinical presentation of a patient, but the context of the care environment to which they present. Understanding and improving factors that affect streamlined workflow, such as provider or department busyness or experience, are essential to improving these care processes, but have been difficult to measure with traditional approaches and clinical data sources. In this exploratory data analysis, we aim to determine whether such contextual factors can be captured for important clinical processes by taking advantage of non-traditional data sources like EHR audit logs which passively track the electronic behavior of clinical teams. Our results illustrate the potential of defining multiple measures of contextual factors and their correlation with key care processes. We illustrate this using thrombolytic (tPA) treatment for ischemic stroke as an example process, but the measurement approaches can be generalized to multiple scenarios.

1. Introduction

The context in which medical care is provided can be as important as clinical factors in determining outcomes from disease^1–3. The time of day, busyness of the emergency department, cognitive burden on specialists, their experience with similar cases or transitions of care can all have a large impact on the ability to provide appropriate care for our patients^4–6. This is especially so for time-sensitive conditions like myocardial infarction or stroke, where the timing of interventions and ability to work with consultants may acutely affect the delivery of care and thus likelihood of a positive outcome for the patient. Such contextual factors are typically not well-represented in traditional (electronic) medical data sources however, limiting our ability to measure, understand, and improve them.

We will focus on emergency treatment of ischemic stroke as an example scenario, as morbidity and mortality highly depends on the timeliness of diagnosis thrombolytic therapy.^7,8 For patients with stroke conditions, the treatment team needs to quickly determine the type of stroke (ischemic or hemorrhagic) by means of physical exams and CT or other head imaging, and whether or not they are a candidate for thrombolytic therapy through history, medication review and lab studies. If the patient is identified as having an ischemic stroke, tissue plasminogen activator (tPA) is an effective medication that can break up a blood clot and treat the patient safely in most cases within 4.5 hours after the symptom onset. Safe and efficacious treatment with tPA is highly time-sensitive and quick treatment not only enhances the chances of survival but also reduces after-stroke disabilities.⁷ Thus, management of stroke relies on fast, accurate diagnosis, coordination, and execution of care processes. However, many factors may affect these processes, such as the busyness of the emergency department and individual providers or limitations in the lab or imaging facilities.

Healthcare quality organizations like the Agency for Healthcare Research and Quality (AHRQ) have respectively sought methods to evaluate the effect of complex systems on the delivery of care. They use process measures like the above stroke “time-to-tPA” to determine how well an institution is hitting benchmarks that connote quality care⁹. Process measures measure the specific steps in a process that lead to a particular outcome or metric of interest. Despite their lack of focus on the specifics of these clinical issues, they are believed to capture the totality of the experience¹⁰. According to the Agency for Healthcare Research and Quality, “While outcome measures may seem to represent the “gold standard” in measuring quality, an outcome is the result of numerous factors, many beyond providers’ control.”

As we attempt to make targeted improvements and learn from these issues, a more granular approach may be necessary. Each process measure may in fact be made up of several other processes or be affected by various elements of the care environment. We may want to know when clinicians undertook certain tasks, how often they were interrupted or who they were working with. Traditionally, this requires labor-intensive primary data collection methods such as time-motion studies, surveys, or interviews. These methods can be limited in scale and do not even always succeed in capturing the variability in these tasks over time¹¹.

The continued adoption of Electronic Health Records (EHR) progressively increases the amount of clinical data and documentation that are naturally captured in computer systems. A relatively underutilized data stream captured is the audit (also known as the event- or access-) log that tracks who is logged in, where they are, and what actions are they performing in the EHR. The primary purpose of such data is to maintain the security of protected health information as required by HIPAA, but they also represent a byproduct of users’ granular interactions with the medical record. This includes, but is not limited to, ordering of medications, acknowledgment of those orders, communications between providers, scanning of blood products, bed requisitioning and movement of patients as well as the time stamp of all of these actions.

The audit log has thus been promoted as a new window into clinical care processes which may be utilized for health services research^11,12 in ways that would previously have been infeasible. Audit log data has been used to study clinical workflows, transitions of care and provider responses to alerts^13–18. Most prior studies have used the access log data to identify clinical workflow patterns¹⁷.

However, this data has not yet been used to measure the effect of contextual factors on clinical processes. Our exploratory data analysis objective is thus to determine whether audit log data can be used to illuminate contextual factors that affect clinical processes, using diagnosis and treatment of ischemic stroke as an example. We aimed to develop measures for “busyness” and “team experience” and assess their relationship with time-to-tPA, a primary process measure for improving stroke outcomes when presenting to the ED.

2. Methods

2.1. Defining the Cohort

Our cohort consists of Stanford Health Care Hospital patients aged 18 years and older who presented to the ED from January 1, 2010 through December 31, 2018 and received tPA within 4.5 hours of presentation. 4.5 hours is the standard window for time from onset of symptoms to safely giving tPA as described by the American Stroke Association (ASA). Thus, we chose 4.5 hours as our window to allow for abrupt onset of symptoms at time of arrival to ED through the end of that window. We suspected that if presenting after 4.5 hours, the patient would thus not be eligible for our outcome of interest (time-to-tPA) or may not have been a presentation to the ED setting.

The outcome of interest for this cohort is the time-to-tPA administration. Given the many audit log actions that may occur over an entire encounter, which may last days or weeks for a stroke patient, we might consider restricting our analysis to those events that occurred within the diagnostic window (presentation through tPA administration). However, we recognize that the events before patient presentation may have an affect on the context of the care environment. Furthermore, we understand that not all actions that occur during the diagnostic window are recorded in real-time at the terminal, and some may be recorded after the fact (i.e. note-writing or logging medication administration). Thus, in our study, we included those audit log events that occurred within a window 60 minutes before and after the tPA administration time.

2.2. Defining Contextual Features

We define various types of “busyness” and “team experience” features based on the audit log data for our patient cohort and investigate their association with time-to-tPA administration. Audit log data captures many different types of activities regarding patients’ clinical information, demographics, workflow, reporting, billing, etc. The top 8 most frequent audit log activities are represented in Figure 1. Note that these activities include both clinical and non-clinical activities.

Figure 1:

Audit log data captures any actions such as chart review, edit, reporting, etc related to the patients. These actions can be grouped into one the 8 activity groups shown in this figure.

We used these distinctions between types of audit log events to define our sub-types of busyness and experience. For example, we define the department-level busyness in terms of several audit log action categories; e.g. all clinical and non-clinical actions, only clinical actions or only imaging actions. Below, we define several terms which are the bases of our contextual features.

Time-to-tPA: The time from a patient’s presentation until their tPA administration time.

Diagnostic window: The window 60 minutes before and after the tPA administration time.

Clinical action: Any action (event) recorded in the audit log data which is directly related to clinical care processes such as editing or reviewing the patient’s encounter information, flowchart, reports and notes, orders, medications, labs or imaging results.

Active patient: An active patients is one with an associated clinical action in the audit log data recorded within a fixed time-window.

Treatment team: The treatment team for a patient consists of all of the providers who access the patient’s clinical information or perform any clinical action.

2.3. Provider/Department Busyness

Busyness can take many forms. Cognitive load, movement throughout a department, new patients coming in the door or number of patients who are critically ill in the department might all be considered elements of busyness for a provider or department in general. But individual providers may not be affected by the total department or team busyness in the same way, and each subsequent encounter they manage may be reflected in department, team or individual busyness. In this section, we introduce and analyze the general department-level and provider-level busyness in two separate categories. We analyzed the association of each of the extracted factors with respect to time-to-tPA using a simple linear regression model.

2.3.1. Department-Level Busyness

The department-level busyness attempts to capture the total work-load of a department at a given point in time. This can be thought of as the relative amount of work by all providers on all the patients in the same department (normalized by the number of providers). In general this implies how busy the treatment department is during the diagnostic window. In the following we introduce several department-level busyness features.

Total number of actions: This factor measures the total number of actions recorded in the audit log data during the patient’s diagnostic window, normalized by the total number of providers. This measure is agnostic to the type of action and rather reflects the total amount of activity at a terminal at a given time. We recognize that work at a terminal can be a cognitive burden and so here attempt to be inclusive of all actions performed.

Total number of clinical actions: Some actions at a terminal are more directly related to clinical care than others. For example, writing an order for a medication or imaging study is likely more indicative of clinical busyness than, say, logging in or simply opening a chart. This factor measures the total number of clinical actions recorded in the access log data during the patient’s diagnostic window, normalized by the total number of providers. These actions include editing or reviewing the patient’s encounter information, flowchart, reports and notes, orders, medications, lab and imaging results, etc.

Total number of imaging actions: In cases like stroke, the ability to order, complete and review imaging is a time-sensitive condition. The ability to perform an imaging modality is dependent on how busy the radiology technician, transporter or radiologist is. This factor measures the total number of imaging actions recorded in the access log data during the patient’s diagnostic process, normalized by the total number of staff and clinicians involved in imaging actions. These actions include ordering, uploading or reviewing an imaging procedure for all patients.

Total number of active patients: At any given time in a department, some patients require more acute interventions, actions, orders or reviews. Some patients may be undergoing acute workups while others may be boarding for hours while awaiting transfer or discharge. This factor aims to measures the total number of active patients by including those with any audit log activities during the stroke patient’s diagnostic window, normalized by the total number of providers.

2.3.2. Provider-Level Busyness

While the busyness of a department as a whole likely affects an individual provider’s busyness therein, the two are not necessarily proportional. Some providers may have more patients undergoing active workup than others. The provider-level busyness metrics attempt to assess the work-load of each individual provider in the treatment team during the stroke patient diagnostic window. In the following we introduce several provider-level busyness features.

Number of clinical actions: As above, we capture the total number of any clinical actions each of the providers in the treatment team perform during the patient’s diagnostic window. These actions include adding or reviewing the patient’s encounter information, flowchart, reports and notes, orders, medications, lab and imaging results, etc. The final measure for a patient is the average of all of the individual provider busyness scores in the treatment team.

Number of process-specific clinical actions: Some of the clinical actions such as ordering labs, images and medication as well as reviewing the results are more critical in timeliness of the stroke treatment process. Thus, here we investigate the impact of the busyness of the providers in terms of these specific clinical actions on the time-to-tPA. For each provider in the treatment team, we capture the total number of these actions during the patient’s diagnostic window. The final measure for a patient is the average of all of the individual provider process-specific actions of their team.

Number of active patients: Some providers may be managing multiple patients at once, while others may have only the stroke patient to focus on when they arrive. This factor measures the total number of active (as measured above) patients assigned to a provider during the stroke encounter’s diagnostic window. Again, for each patient, the final measure is the average of the individual provider scores in that stroke encounter treatment team.

2.4. Treatment Team Experience

Prior experience managing a disease or working with individual treatment team members may affect the efficacy and timeliness of coordinating and executing care. In this section we define several factors which may reflect different individual or shared provider experience.

Individual Provider Experience: For each provider, we measure the number of prior tPA-treated ischemic stroke cases from our cohort on which an individual provider has been a member. The associated score is the average of the experience scores for all of the providers in the treatment team for the particular stroke encounter.

Time Since Last Experience: We expect that proximity to a recent case may affect how quickly a provider responds to an acute stroke. Respectively, we measure the time since a provider in the treatment team has last experienced a tPA-treated ischemic stroke case from our cohort. The associated score is the average of the recent experience scores for all of the providers in the treatment team for that encounter.

Treatment Team Shared Experience: Providers do not manage acute strokes in isolation. Each member of the team has a specific role and responsibility, which may improve with experience. Though providers may share experiences across types of patients and clinical presentations or across departments, here we focus on the shared experience of the providers in the treatment team for prior tPA-treated stroke cases from our cohort. Specifically, we measure the number of times a pair of providers in the treatment team has experienced a tPA-treated ischemic stroke case through network combination. The associated score is the average of the shared experience scores for all of the provider pairs in the treatment team for that encounter. Here’s a very simple example for computing the shared experience score: Assume that the treatment team for a patient consists of three providers P1, P2 and P3. Let the shared experiences for the provider pairs (P1, P2), (P1, P3), and (P2, P3) are respectively 2, 3 and 1. Then the associated patient score is the average of the provider pair scores, which is 2.

2.5. Association with Time-to-tPA

With the above contextual factor measurement proposals, an important question that arises is whether any of them individually or in combination is associated with time-to-tPA as an indicator of diagnostic and execution efficiency. We assessed the association between each of the proposed contextual factors and time-to-tPA using a linear regression model with 95% confidence interval, the Pearson correlation coefficient and p-values.

We also explored the multivariate association of the contextual factors and time-to-tPA. For this purpose, we trained a model (multivariate function) using the contextual features that minimizes the mean square error (MSE). To reduce the risk of model overfitting, we employed feature selection based on the highest correlation coefficients. Further, we used support vector regressor (SVR) as a regression model with smaller degrees of freedom (to account for the relatively small data size). We considered various kernels such as linear, polynomial (several degrees of freedom) and radial basis function (RBF) kernels. We applied a 5-fold cross-validation technique with mean absolute error (MAE) metric to evaluate the performance of the regression model. At each cross-validation iteration, we respectively use 70% and 30% for training and validation.

3. Results

There were a total of 269 patients in out cohort, with 140 female and 129 male genders. The majority of the patients (80%, 214/269) were aged 60 or older.

Context Correlation: Using the audit log of the Stanford University Epic EHR, we were able to extract 10 features related to the context of medical care provided in the ED at the department, team and individual provider levels defined in section 2.2. We investigated the correlation of our proposed busyness and experience features with time-to-tPA. Time-to-tPA had an average value of 55.0 minutes with min and max values 1 and 218 minutes. Figure 2 represents the association between department busyness in terms of number of imaging actions (normalized per the related providers) and time-to-tPA. We fit a linear regression model with 95% confidence interval and along each axis we plotted the probability distribution function of the variables. The measured correlation coefficient and associated p-values are respectively −0.03 and 0.65, which shows that the busyness in terms of imaging actions is not correlated with time-to-tPA. The results are summarized for the rest of the features in Table 1 and Figure 3. The measured correlation coefficient and associated p-values are shown in Table 1.

Figure 2:

Association between department busyness in terms of number of imaging actions (normalized per the related providers) and time-to-tPA. We also fit a linear regression model with 95% confidence interval. Along each axis we also plot the probability distribution function of the variables. The measured correlation coefficient and associated p-values are respectively −0.03 and 0.65.

Table 1:

Busyness and treatment team experience features and their correlation with time-to-tPA

Feature Type	Contextual Measure	Correlation Coefficient	p-value
Department busyness	Number of all actions	0.20	0.001
Department busyness	Number of clinical actions	0.14	0.028
Department busyness	Number of imaging actions	-0.03	0.670
Department busyness	Number of active patients	0.17	0.006
Provider busyness	Number of clinical actions	0.05	0.476
Provider busyness	Number of specific actions	-0.03	0.656
Provider busyness	Number of patients	-0.10	0.101
Treatment team experience	Individual provider experience score	-0.25	10−5
Treatment team experience	Shared provider experience score	-0.29	10−6
Treatment team experience	Time since the last tPA Case	0.04	0.503

Figure 3:

Summary of the association results between the time-to-tPA and department busyness (left column), provider busyness (middle column), and treatment team experience (right column).

Multivariate Correlation: Above, we showed that several busyness and experience contextual factors individually have moderate correlation with time-to-tPA. Here, we selected the three most important features based on the correlation scores in Table 1. According to this table, the three features with the highest correlation coefficients are number of all actions, individual provider experience score and shared provider experience score. The correlation results between the trained multivariate function and time-to-tPA are represented in Table 2. An SVR model with a linear kernel gives the strongest multivariate correlation compared to polynomial and RBF kernels.

Table 2:

Multivariate correlation of the contextual factors and the time-to-tPA using different models

Kernel	Linear	Polynomial (df=2)	Polynomial (df=3)	Polynomial (df=4)	Polynomial (df=5)	RBF
Correlation Coefficient	0.44	0.31	0.29	0.33	0.32	0.09

4. Discussion

EHR audit logs can be used to define multiple contextual factors around care processes such as busyness and experience. In our example case of emergency treatment for ischemic stroke, most of our example contextual measures show some correlation with time-to-tPA. Among the proposed department-level busyness measures, the total number of actions had the strongest correlation with time-to-tPA. A greater total number of all actions undertaken in the ED during the treatment window was associated with longer time-to-tPA.

Two other department-level features, the total number of clinical actions and number of active patients, were also positively correlated with time-to-tPA, though to a lesser degree (Table 1 and Figure 3). This may be due to the fact that not all clinical activities (like evaluating the patient, talking to family members or discussing the presentation with other members of the medical team) are captured in the EHR. However, this distinction illuminates that further work may be necessary to determine which activities contribute most to busyness or distraction from patient care.

Beyond overall department workload, we measured the busyness of individual providers involved in the treatment of stroke patients. The provider busyness was measured in terms of two subgroups of actions: all clinical actions and more specific clinical actions. Both showed smaller correlations with time-to-tPA compared to similar measures at the department-level.

Somewhat counter-intuitively, we found that the greater number of patients at the department or individual level during the treatment window was correlated with faster time-to-tPA ordering. We presumed that more patients and workload in a department may make it more to difficult to focus on timely care processes for any particular patient, but this was not directly supported by our results. An alternative clinical interpretation is that when the ED is busy, providers have less time to tease out the nuance of a case and ironically become more ”efficient” by resorting to algorithmic management and immediate imaging. Thus, process measures like time-to-tPA might actually improve in the setting of a busy treatment team, though it is unclear if this impacts clinical or patient-centered outcomes or possibly medical errors and physician burnout.

Similarly, some providers had anticipated that a busyness at the radiology department level, in terms of how many imaging studies were being performed during the diagnostic window, would be an important factor in time-to-tPA. It was thought that more radiology studies being performed would stress the radiology department, techs and machines may be busy, meaning that there might be delays to getting stroke patients into the CT scanner. However, our data did not reflect this (Figure 2). There seemed to be no correlation between the number of radiology events in the audit-log and the time-to-tPA. However, given that an available CT scanner is recognized to be a critical action for stroke, most departments have a protocol in place to clear the CT scanner as soon as a possible stroke patient has been identified and a ”code stroke” called. In fact, given that this occurs for each stroke in our cohort, a lack of correlation with increased or decreased time-to-tPA implies that the processes utilized by the radiology department are working appropriately. Whether the radiology department is busy or not, there should be negligible effect on time-to-tPA.

We found that individual and team historical experience was most strongly correlated to the time-to-tPA. With respect to the individual experience, the more tPA cases a provider had managed, the faster tPA was likely to be ordered as seen in Figure 3 (right column). There are many possible reasons for this correlation. Perhaps, providers who have seen more strokes are better able to identify the cardinal features of the conditions and thus faster to make the diagnosis and place orders. Although it could also be due to practical considerations like the ease of finding an order set, the ability to review past history or even ability to review imaging results. These skills take time to master, but all of which are likely to improve with experience.

A stronger correlation was found for the shared tPA experiences of the team. An increase in shared experiences of team members treating stroke patients with tPA together related to a decreased time-to-tPA in subsequent cases. The care of stroke patients does not happen in isolation. It requires multiple specialties and team members from the emergency, neurology and radiology departments and across provider types like physicians, nurses and technicians. Thus, more shared experiences across providers may help streamline the many forms of communication and task completion which need to be completed for timely care. This may represent and opportunity to prospectively evaluate the scheduling of treatment teams. A department may use this information to guide staffing decisions or provide education or simulation opportunities to members who have not worked together, which may help improve process orientation and coordination.

Improved inference about time-to-tPA requires considering the information about a group of the contextual factors rather than simple individual feature information. Thus, compared to each context correlation result, a multivariate function of a subset of features shows a stronger correlation with time-to-tPA (Table 2), implying that no single factor can account for the complete context of care. Further, the results show that the association between the selected features and time-to-tPA is better modeled by a linear monotonic relation than other types of polynomial and radial basis functions. Our multi-variable correlation models illustrate the potential predictive power of these contextual factors, but further studies will be necessary to determine the strength of these correlations when accounting for additional clinical features.

We believe that there are many similar time-sensitive processes in medicine which may benefit from evaluation of audit log data. The management of acute coronary syndrome through angioplasty or sepsis via antibiotics are particularly suited to our methodology. While time to head CT and tPA may be stroke specific, features like department busyness or provider experience are not. However, our preliminary definitions of busyness and experience likely do not completely account for these dynamic concepts and differ between departments, specialties or roles. There may be more sensitive or specific definitions that can be developed in future studies for different care settings, but this study importantly illustrates this potential to do so with audit log data in a way is largely not possible at all with traditional data sources.

The audit log offers some unique challenges as well. We found that documentation about types of provider may not be complete or may not account for their state of training (ie. first-year resident versus third-year) meaning that these features must be inferred from other data. Furthermore, a provider may appear ”new” to an institution but may be familiar with their particular EHR and may thus adapt more quickly to unique order sets or processes. On the other hand, adept clinicians may have practiced at an institution for many years but have delays in ordering or providing care when a new EHR is adopted or a new hospital opened. We did not evaluate these specific subtypes of providers in this exploratory evaluation, however, which may thus have affected our interpretation of provider experience on time to tPA.

Furthermore, while audit-log data may be consistent within an institution, that may not be the case between institutions - even if they use the same EHR vendor. Thus, extraction of variables of interest may represent a barrier given that audit log data may be recorded uniquely.

While a cohort of 269 cases of stroke treated with tPA does give insight into the nature of stroke management within one hospital system, generalizations with regard to the context of care will depend on planned future work to replicate the study across multiple institutions. Additionally, in our study, individual cases were not controlled for clinical factors that may also have led to differences in care or treatment time.

Conclusion

In an exploratory data analysis of EHR audit logs, we illustrate the potential to measure contextual factors of busyness and experience in key care processes for the example scenario of thrombolytic treatment for ischemic stroke. Treatment team and individual experience managing similar stroke cases and the total number of actions undertaken in the ED during the diagnostic window were found to have the strongest correlations to time-to-tPA. This methodology can open an important window into the otherwise hidden context of clinical environments that impact the quality of care.