Using Visual Analytics to Determine the Utilization of Preoperative Anesthesia Assessments

Summary

Background

Preoperative assessments are a required and essential element of anesthetic care, yet little is known about the utilization of these documents by clinicians who are not part of the anesthesia care team. As part of perioperative workflow restructuring, we implemented a data visualization technique of electronic medical record audit log data to understand the utilization of preoperative anesthesia assessments by non-anesthesia personnel.

Methods

An audit log cache containing 140 days of data was queried for all accesses of preoperative anesthesia assessment documents for any patient who had a preoperative anesthesia assessment that was accessed during that period. User roles were aggregated into categories. Descriptive statistics and data visualization were generated using R (R Software Foundation, Vienna, Austria). Comparisons were performed with the Wilcoxon signed rank test with continuity correction.

Results

During the study period, 73 802 (0.015%) of the 485 062 902 audit log accesses were preoperative anesthesia assessments representing 412 departments, 302 user roles, and 3 916 distinct users who accessed preoperative anesthesia assessments from 14 235 surgical cases. Each assessment was accessed 2.9 times on average. Assessments performed in the preoperative anesthesia assessment clinic were accessed more frequently than those created on the day of surgery in the preoperative holding room (3.58 ± 5.18 v. 1.98 ± 1.76 average views; p<0.0001). We observed accesses of these documents by pathology and general surgery researchers, as well as orthopedics attending physicians accessing documents that were two years old.

Conclusions

This approach revealed patterns of utilization that had not been previously identified, including usage by surgical residents, surgical faculty, and pathology researchers both before and after the surgical event for which the documents are generated. Knowledge of these dependencies directly informed perioperative workflow restructuring efforts. This visual analytic approach could be broadly utilized to understand documentation dependencies in a variety of clinical contexts.

1. Introduction

Preoperative assessment of surgical patients by a clinic-based anesthesia team in advance of elective surgery has been demonstrated to reduce case cancellations on the day of surgery, improve the efficiency of clinical operations and lead to patient satisfaction [1–3]. These benefits are largely seen with complex patients or those undergoing more invasive surgeries, as healthier patients undergoing minor surgery can typically be evaluated on the morning of surgery without an increased risk of cancellations or delays [4]. This assessment is sometimes the only comprehensive history and physical that a surgical patient receives and can be a reliable source of patient information. As part of a perioperative workflow restructuring initiative at our institution, we evaluated closing our preoperative clinic which performs preoperative history and physical exams, and replacing it with a system where preoperative history and physical exams would not be completed until immediately prior to surgery. Thus comprehensive patient information would not be available prior to the day of surgery, with unknown consequences on workflow. Prior to making this shift, we sought to assess the current utilization of this information.

As part of the Health Information, Privacy and Portability Act, health care institutions are required to maintain audit log data that record all accesses of electronic medical records [5]. In addition to meeting this requirement, these data have been used for other purposes as well, including understanding the life cycle of electronic medical record documents [6], exploring trainee utilization of the electronic medical record [7], and for medicolegal purposes [7, 8], in addition to their primary purpose of addressing patient privacy concerns. As a consequence, these logs represent a detailed, rich source of information that could describe how electronic medical record documents are actually utilized within an institution, and is a data source that is required by law to be maintained. Thus, these logs are well-suited to inform workflow analysis.

Visual analytics in healthcare have been used to deal with the information overload that is encountered when dealing with large volumes of data generated by electronic medical records [9]. Newer techniques have been developed to look at individual patients, as well as larger populations, over time [10, 11]. In recent history, visual analytics has even been utilized to optimize drug-drug interaction alert rules [12] and model patient severity in intensive care units [13]. Audit logs are no exception to this information overload challenge, as multiple audit log records can be generated by every interaction that personnel have with an electronic medical record. Techniques have been described to filter through this data source to identify accesses that may be concerning from a privacy perspective [14, 15]; however, these algorithms are not suited to inform workflow analysis.

2. Objectives

Few data exist describing the utilization of routinely generated electronic clinical documentation even though 67% of US academic hospitals reported use of automated electronic anesthesia records in May 2013 (16). These electronic records have the potential to serve as a huge reservoir of information for perioperative patient care. In this article, our goal is to understand the utilization of preoperative anesthesia assessments by non-anesthesia personnel, to leverage audit logs to inform workflow restructuring, and to demonstrate how visual analytics can be used to efficiently identify patterns within large data sets.

3. Methods

The Institution Review Board of the Vanderbilt University Medical Center approved this study. Additionally, approval was sought and obtained for this analysis from the home institution’s privacy office. The audit logs for our electronic medical record were queried from the audit log cache in our data warehouse, which contains 140 days of data. We queried that source for all accesses of our preoperative anesthesia assessment documents between January 24, 2013 and June 13, 2013. While the audit log cache contains 140 days of data, each preoperative assessment is retained permanently as part of that patient’s EMR (Electronic Medical Record). Therefore, during the 140-day cache period it is possible to see accesses of documents of any age. We included all patients who had a preoperative anesthesia assessment that was accessed between these dates.

For each user associated with an audit log access, we determined the user’s role and department assignment. Due to privacy concerns, we only analyzed these accesses in aggregate by user role and department without the identification of individual users. We identified users who worked in the preoperative holding room area, post-anesthesia care unit, operating room, preoperative anesthesia assessment clinic and anesthesia department, and excluded their accesses from analysis. These users, again, were identified by their role and department assignment.

Using a unique identifier associated with these documents, we determined the day of surgery and calculated the interval in days between the electronic medical record access of the preoperative assessment and the day of surgery, which we termed access day relative to surgery. Finally, we determined if the preoperative assessment occurred in a preoperative clinic visit or on the day of surgery in the preoperative holding room.

To develop a data visualization plot of the audit log accesses, we determined the top 30 departments by total number of accesses and created user role groups. User roles were aggregated into seven categories: nurse practitioner, faculty, coding, case manager, research, resident, or registered nurse. Individual accesses were visualized for the top departments by displaying the access day relative to surgery, grouped by department and color-coded by user role group. Two plots were generated: one that displayed all points to demonstrate all usage patterns, and a second with a point transparency setting of 90% to provide a visualization that accounted for overplotting. A color palette was selected that allows for full interpretation for those with red-green color vision deficiency. Descriptive statistics and data visualization were generated using R, using the ggplot2 package (R Software Foundation, Vienna, Austria). Normality was assessed using the Kolmogorov-Smirnov test, and non-parametric comparisons were performed with the Wilcoxon signed rank test with continuity correction. An α level of 0.05 was taken as reference to detect statistical significance in all analyses.

4. Results

During the study period, 485 062 902 audit log accesses were identified over 140 days. Of these, 73 802 (0.015%) were accesses of preoperative anesthesia assessments. After removing accesses associated with anesthesia personnel, in the 41 036 accessed that remained, we identified 412 departments and 302 user roles. We identified 3 916 distinct users who accessed preoperative anesthesia assessments from 14 235 surgical cases. Each assessment was accessed 2.9 times on average. Assessments performed in the preoperative anesthesia assessment clinic were accessed more frequently than those created on the day of surgery in the preoperative holding room (28 355 vs 12 681 total views; 3.58±5.18 vs 1.98±1.76 average views; p<0.0001).

We analyzed the user roles that accessed the preoperative anesthesia assessments the most and determined the most frequent access day relative to surgery (►Table 1). This information was supplemented by our data visualization (►Figure 1), which together revealed several patterns of utilization described below.

Table 1.

Most frequent utilizers of preoperative anesthesia assessments, by distinct user role and department, listed with total count of accesses and most frequent difference between access day and day of surgery.

User role	Department	Access count	Most frequent difference between access day and day of surgery
Coding	Medical Information Systems	4 526	5
Nurse practitioner	VUH Nurse Practitioners	2 960	0
Resident physician	General Surgery Housestaff	1 889	0
Clinical documentation coordinator	Medical Information Services	1 784	1
Research	Pathology, Microbiology and Immunobiology	1 597	-1
Resident physician	Urology Housestaff	1 559	0
Health data coordinator	Medical Information Systems	920	1
Quality consultant	Center for Clinical Improvement	770	116
Medical records analyst	Medical Information Systems	731	5
Attending physician	Orthopedics and Rehabilitation	713	737
Coding	Surgery/Trauma	662	10
Resident physician	Neurosurgery Housestaff	552	0
Registered nurse	Orthopedics	552	0
Tumor registrar	Cancer Registry	536	252
Coding	VMG Coding	498	2
Nurse practitioner	VHVI Clinical Support	408	-1
Resident physician	Ob/Gyn Housestaff	402	0
Performance improvement analyst	Center for Clinical Improvement	400	110
Coding	Neurosurgery	378	7
Registered nurse	Gyn Surgery	374	0

Fig. 1.

Data visualization

The user role associated with the most accesses was coding personnel, who accessed these documents most frequently 5 days after surgery (►Table 1). This was a known usage of preoperative anesthesia assessments, as these documents are checked prior to anesthesia billing and to supplement comorbidity coding for hospital billing. Surgical residents and surgical nurse practitioners frequently accessed these documents on the day of surgery from departments including general surgery, which was also expected. However, we observed accesses of these documents from pathology researchers, which occurred most frequently the day prior to surgery (black oval, ►Figure 1).

Similar access patterns were found from surgical nurse practitioners, residents, and attending physicians from other surgical departments. We saw a large cluster of general surgery researcher accesses of documents created 5 years previously (red oval, ►Figure 1), and a similar cluster from orthopedics and rehabilitation attending physicians accessing documents that were two years old (blue oval, ►Figure 1). In order to highlight the most frequent access patterns, we additionally visualized these data with partial transparency to account for overplotting (►Figure 2).

Fig. 2.

Visualization of data with partial transparency to account for overplotting.

5. Discussion

In this study, we demonstrated the use of visual analytics to examine a large data set of audit log access records. We found that despite these documents being authored by anesthesia teams, they are frequently accessed by non-anesthesia personnel representing a diverse set of departments and user roles. In conjunction with descriptive statistics, this approach identified several previously unrecognized uses for preoperative anesthesia assessments. Using only descriptive statistics, key pieces of information would have been overlooked such as pattern of use over extended periods of time. Using visual analytics, the use of anesthesia preoperative assessments 2–5 years after their creation became readily apparent. Additionally, the use of this visual approach made it easier to identify that the personnel viewing the records changes in an interesting way over time. Analyses such as this one demonstrate the value of audit logs in understanding workflow, and the specific role that visual analytics can have in identifying patterns within large data sets.

Prior to the results of this analysis, our institution had planned to restructure perioperative workflow to shift patients from preoperative clinic assessment to assessment on the day of surgery. During planning meetings with surgical department heads, it was stated that documents authored by anesthesia teams were not used by members of the surgical departments as part of their workflow. Quantitative analysis revealed that this was not the case, but rather these documents were frequently accessed by a wide variety of non-anesthesia personnel. Most likely, these documents were accessed to provide a comprehensive understanding of patients’ medical history, although this is speculation. Following the completion of this study, the departments involved decided to abandon the proposal to close the preoperative clinic. Instead, the preoperative evaluation model was shifted. While the majority of visits are performed as in-person assessments, a portion of the history and physical assessments were converted to histories only, and a small fraction of patients – those without significant past medical history – are evaluated through day of surgery assessments. The type of assessment that each patient receives depends on patient comorbidities and type of procedure, and is specified through VPEC (Vanderbilt Preoperative Evaluation Center) protocols that were developed. Additionally, given the demonstrated need for preoperative information, the timing of assessments relative to surgery was altered. Assessments are now performed well in advance of surgery to allow adequate time for any interested party to review them prior to the day of surgery.

Previously published work related to audit log accesses has largely related to identifying patterns suspicious for privacy violations [14, 15, 17]. Although audit logs have been used to understand care teams related to obstetric and neonatal care [18], to these authors’ knowledge they have not previously been utilized to inform workflow restructuring.

This study has several limitations. While we are able to track access of electronic medical record documents, there is no mechanism to determine the value that viewing these documents gave to individual users. It is possible that these documents were accessed but not read. Additionally, because we analyzed these data in aggregate only due to privacy concerns, we were not able to contact users to determine why they were viewing these documents, nor were we able to perform detailed analyses on the clusters identified. Finally, it is a common practice at our institution for clerical staff to print anesthesia preoperative assessments the evening prior to surgery when available to include them as a part of the patient’s surgical paper chart. As it is not possible to track usage of these paper copies, our analysis may underestimate the utilization of these documents. However, these paper copies are typically used by anesthesia personnel and perioperative nurses that were excluded from this analysis, minimizing the impact of any underestimation.

6. Conclusion

This study enabled a better understanding of the utilization of preoperative anesthesia assessments by non-anesthesia personnel, provided audit log information to impact workflow restructuring, and demonstrated how visual analytics can be used to efficiently identify patterns within large data sets. As maintaining audit logs is required by law, this approach represents one which could be broadly utilized in a variety of clinical settings and for various administrative and clinical purposes.