Characterizing the macrostructure of electronic health record work using raw audit logs: an unsupervised action embeddings approach

Sunny S Lou; Hanyang Liu; Derek Harford; Chenyang Lu; Thomas Kannampallil

doi:10.1093/jamia/ocac239

. 2022 Dec 8;30(3):539–544. doi: 10.1093/jamia/ocac239

Characterizing the macrostructure of electronic health record work using raw audit logs: an unsupervised action embeddings approach

Sunny S Lou ^1,², Hanyang Liu ³, Derek Harford ⁴, Chenyang Lu ⁵, Thomas Kannampallil ^6,^7,^8,^✉

PMCID: PMC9933072 PMID: 36478460

Abstract

Raw audit logs provide a comprehensive record of clinicians’ activities on an electronic health record (EHR) and have considerable potential for studying clinician behaviors. However, research using raw audit logs is limited because they lack context for clinical tasks, leading to difficulties in interpretation. We describe a novel unsupervised approach using the comparison and visualization of EHR action embeddings to learn context and structure from raw audit log activities. Using a dataset of 15 767 634 raw audit log actions performed by 88 intern physicians over 6 months of EHR use across inpatient and outpatient settings, we demonstrated that embeddings can be used to learn the situated context for EHR-based work activities, identify discrete clinical workflows, and discern activities typically performed across diverse contexts. Our approach represents an important methodological advance in raw audit log research, facilitating the future development of metrics and predictive models to measure clinician behaviors at the macroscale.

Keywords: raw audit logs, unsupervised learning, clinical workflow

INTRODUCTION

Over the past 20 years, electronic health record (EHR) adoption has increased 10-fold.¹^,² Studies have shown that over 50% of a physician’s time is spent interacting with the EHR,3–5 as it plays a central role in facilitating the planning, execution, documentation, and management of clinical care. Given the significance of the EHR in physicians’ work activities, and its contributory role toward documentation burden and burnout,6–8 it is important to study how physicians use the EHR to identify opportunities for improvement. However, traditional methods to study physician workflow, such as observational and time-motion studies, are limited because they are labor intensive and susceptible to inter-observer variation. EHR audit logs, which automatically capture click activities performed within the EHR, offer an opportunity to study physician EHR behaviors reliably and reproducibly at scale.⁹^,¹⁰

Much of the recent audit log research on physician behaviors has relied on vendor-provided metrics (eg, Epic Signal, Cerner Advance).¹¹ Although useful for summarizing clinician EHR usage on an aggregate basis,¹² such vendor-derived metrics are limited in their lack of temporal granularity and customizability.¹³ In contrast, raw audit logs—mandated by the Health Information Portability and Accountability Act—unobtrusively capture a trail of clinician activities within the EHR, affording researchers the methodological flexibility to study fine-grained physician behaviors at the microscale. However, raw audit logs lack context for clinical activities and are difficult to interpret in isolation. For example, although clinicians use the EHR for performing specific high-level clinical tasks (eg, writing a note), these tasks generate multiple audit log actions (eg, reviewing a previous note, reviewing laboratory results, opening a note template, using smart text, signing a note). Raw audit logs capture each EHR action at a high level of granularity such that the high-level task is often not apparent, especially if multiple activities are simultaneously completed. In addition, the names of EHR actions in the raw audit log files are often not easily interpretable.

Although efforts have been made for manual annotation of audit logs by clinician experts to provide context,³^,⁵^,¹⁴ these efforts are labor intensive, potentially error prone, and not easily scalable to accommodate new clinical workflows or changes in vendor software, nor to evaluate different healthcare systems. In this brief communication, we describe a novel methodological approach to automatically annotate individual audit log actions using action embeddings, a scalable unsupervised machine learning approach, to learn the context and structure of raw audit log activities. We demonstrate how such embedding annotations can provide insights into clinical activities and workflow.

METHODS

Study population and data

This study was conducted at Washington University, a large academic medical center encompassing 2 academic tertiary-care hospitals (Barnes-Jewish Hospital, St Louis Children’s Hospital) and several outpatient care centers in St Louis, MO. Audit log data on EHR use were collected for 6 months (approximately October 2020 to March 2021) from a cohort of intern physicians from Internal Medicine, Pediatrics, and Anesthesiology, who had consented to provide data as part of a previously described study.¹⁴^,¹⁵ This cohort was chosen owing to their relatively homogenous work responsibilities as primary caregivers with similar levels of clinical experience. This study was approved by the institutional review board of Washington University (IRB # 202004260).

During the data collection period, participating intern physicians worked across inpatient and outpatient settings and performed a diverse range of clinical tasks depending on the clinical unit for which they were providing care. These clinical settings included inpatient ward medicine, inpatient consulting services, outpatient primary care, and outpatient subspecialty clinics. Common EHR-based activities included reviewing patient notes, laboratory results, and patient data such as vital signs; ordering medications and laboratory tests; writing notes; and responding to in-basket messages.¹⁴

Raw audit logs for participating physicians were retrieved from institutional databases (Epic Systems, Verona WI). For each audit log action, the timestamp of the action, its descriptive name as assigned by the EHR vendor (ie, “Metric Name”), an identifier for the user performing the action, and the patient identifier were collected. An example raw audit log file is shown in Figure 1A.

Figure 1. — Overview of approach. (A) Example raw audit log before and after augmentation with additional report detail. Vertical line indicates audit log segment illustrated in (B). (B) Illustration for how action embeddings are computed. A 2-layer neural network was trained to predict cooccurring (ie, neighboring) actions for each action using the word2vec skip-gram algorithm. Compared to raw audit logs, detailed audit logs provided a greater diversity of action-neighbor pairs for model training. In this example, only 2 different action-neighbors pairs were observed when raw audit log data were used; in contrast, 4 unique pairs were observed when detailed audit log data were used.

Audit log augmentation

Raw audit logs lacked detail and context for the clinical activities performed. Therefore, we developed additional data queries to provide detail to supplement the raw audit log data. Specifically, we retrieved additional metadata from Epic’s Clarity database for actions related to notes and reports (see Supplementary Material for additional detail), which were used to populate a “Report Name” field indicating the type of report or note that the physician viewed or modified (Figure 1A, rightmost column). For example, although the raw audit log only indicated whether a physician viewed a generic report, additional metadata helped in determining the type of report viewed (ie, vital sign, laboratory results, radiology report). When additional detail was available, “Metric Name” and “Report Name” were used together to represent distinct EHR actions. These supplemented audit logs are henceforth referred to as detailed audit logs.

Training of action embeddings

To derive context and structure from audit log activities, we used embeddings, an unsupervised machine learning technique borrowed from natural language processing (NLP).¹⁶ The goal of embeddings is to learn a numerical representation for tokens (ie, words for NLP, EHR actions for raw audit logs) from a corpus of token sequences such that tokens used in similar contexts—that is, cooccurring with similar adjacent tokens—have similar representations (Figure 1B). This idea is based on the linguistic theory that words (ie, EHR actions) used in similar contexts have similar meanings.¹⁷ In our analysis, each unique EHR action was represented with a distinct token; the specific words contained within the EHR action names were not considered for training the embeddings.

We used the word2vec skip-gram algorithm to learn vector representations for each unique EHR action in our audit log dataset.¹⁸ Raw audit logs were separated into sessions of EHR use—analogous to sentences for words—based on continuous click activities separated by less than 5 minutes of inactivity.¹⁹ A 100-dimensional vector representation for each EHR action was randomly initiated and iteratively refined using a 2-layer neural network to predict the 20 adjacent surrounding actions across all observed sessions of EHR use (Figure 1B). Action embeddings were trained using both raw and detailed audit logs as input; each unique “Metric Name” was used as the action for raw audit log embeddings, whereas each unique “Metric Name-Report Name” pair was used as the action for detailed audit log embeddings. Embeddings were retained only for actions that occurred at least 20 times in the audit log data; unstable representations were more likely below this threshold.²⁰

Assessing workflow diversity

After training, the embedding (ie, 100-dimensional vector representation) of each EHR action contained information on its semantic use and meaning. We measured the pairwise similarity in semantic usage across all EHR actions by computing pairwise cosine similarity scores²¹ using the embedding vectors; these scores range from 0 to 1, with 1 representing the highest similarity (Supplementary Figure S1). By virtue of the embeddings training process, if a pair of actions had a high similarity score, they cooccurred with a similar set of neighboring actions¹⁶; in other words, the 2 actions were frequently used in similar activity contexts. For example, “Results Review accessed” (which indicates that the clinician opened the laboratory results review interface in the EHR) was used by clinicians in a similar context to other actions related to the review of other test result and flowsheet information (Figure 2A). Conversely, 2 actions with a low pairwise similarity score almost never cooccurred with the same neighboring actions and therefore did not share a similar workflow context.

Figure 2. — Pairwise similarity indicates workflow diversity. (A) The top 10 most similar neighbors in embedding space for a representative action “Results Review accessed” are shown, illustrating that the identified actions also correspond to chart review activities frequently performed in similar contexts. (B) For each action, the average similarity of its 10 closest neighbors in embedding space and its frequency (ie, # of occurrences) in the dataset was plotted. A density plot was created, overlaid with histograms illustrating the distribution of frequency and average neighbor similarity. (C) Representative actions with high (>0.9) average neighbor similarity are shown, thus identifying electronic health record (her) activities that are typically part of highly stereotyped workflows. (D) Representative actions with low (<0.6) average neighbor similarity are shown, suggesting that these activities are typically performed across diverse contexts.

We utilized an average neighbor similarity score as a measure of workflow diversity. Specifically, for each EHR action, we identified the 10 EHR actions with the highest pairwise similarity scores with that action (ie, the 10 actions that were used in the most similar context to the source action), and measured the mean of these 10 highest pairwise similarity scores to create an average neighbor similarity score. Actions with high average neighbor similarity were used in highly similar contexts (ie, reproducibly sharing many neighboring actions) with other actions, and thus were likely part of highly stereotyped workflows. Conversely, actions with low average neighbor similarity were not reproducibly utilized in similar contexts and shared few reliable neighbors with other actions, that is, they were used across diverse contexts and workflows. Based on an empirical assessment of the distribution of average neighbor similarity scores (Figure 2B, top histogram), a threshold above 0.9 was used to identify actions with high average neighbor similarity, and below 0.6 was used to identify actions with low average neighbor similarity.

Visualization of the global structure of EHR work

Global similarity of all actions observed in the dataset was visualized using t-Stochastic Neighbor Embedding (t-SNE).²² Each unique action was plotted as an individual data point. Groups representing EHR actions performed in similar contexts were identified by visual inspection of the t-SNE plot. A manual review of the groups for developing descriptive characterizations was performed by a clinician member of the study team.

To provide a comparison to clinician expert manual action annotations, we relied on a previously described set of audit log event categorizations.³^,¹⁴ In this categorization, expert clinicians manually reviewed a list of audit log actions and assigned each action to one of the following categories: chart review, note review, results review, best practice advisories, navigation, order entry, inbox, and note entry. We applied these same categorizations to the audit log events in our dataset and colored the points in the t-SNE plot by category.

All data processing, analysis, and visualization were performed using Python 3.9.10, gensim 3.8.3, and sci-kit learn 1.0.2.

RESULTS

Study characteristics

This study included 15 767 634 audit log actions performed by 88 resident physicians over 6 months. These actions spanned 285 912 sessions of EHR use in diverse clinical workflows across inpatient and outpatient settings. A total of 402 unique “Metric Name” actions were observed in the raw audit log data. After augmenting the raw audit logs with additional report-level detail, 1652 unique “Metric Name-Report Name” actions were observed. The most commonly observed actions in the data are shown in Supplementary Tables S1 and S2.

Characterizing action context and workflow diversity

For each EHR audit log action, workflow diversity was measured using action embeddings and the average neighbor similarity metric. The average neighbor similarity was uncorrelated with action prevalence (Figure 2B). Filtering EHR actions by high average neighbor similarity (ie, average cosine similarity >0.9) enabled the identification of stereotyped, highly specialized workflows such as those used in the emergency department, in the operating room, or on a mobile EHR client (Figure 2C). Conversely, filtering EHR actions by low average neighbor similarity (ie, average cosine similarity <0.6) enabled the identification of activities used across diverse contexts (Figure 2D).

Visualizing the global structure of EHR work

The global semantic structure of EHR use was visualized by plotting the learned embedding vectors in 2 dimensions (Figure 3), thus illustrating the relationships between EHR actions and revealing the EHR use patterns of intern physicians. Groups of actions in this visualization were used to identify specific workflows. Well-defined groups contained activities that cooccurred frequently with each other and infrequently with any other actions, enabling the identification of specialized workflows; for example, use of the mobile EHR client (Figure 3, group 1; Supplementary Table S3) or use of the preoperative assessment module (Figure 3, group 2; Supplementary Table S4). The less well-defined groups of actions represented actions often performed in similar contexts but with greater variability. This enabled the identification of 2 distinct workflows for chart biopsy/patient results review (Figure 3, group 3, Supplementary Table S5 and group 4, Supplementary Table S6), which might be used, for example, during inpatient prerounding or when seeing a new patient in the outpatient setting. These findings highlight the ability of action embeddings to learn clinically meaningful groupings of activities from audit log activities in an unsupervised manner.

Comparing embeddings with clinician annotations of EHR actions

Embeddings provided complementary contextual information about specific EHR actions compared with clinician expert annotations (shown in color, Figure 3). For example, clinician-annotated note review activities were grouped together with result review activities, indicating these activities were commonly performed in similar contexts; in contrast, clinician-annotated note entry and order entry activities were scattered throughout embedding space, suggesting that these actions may not occur in stereotyped workflows but rather across diverse contexts. Thus, although clinician expert annotations were informative regarding the specific granular activity performed, the embeddings contextualized that activity within other related activities and overall workflow.

DISCUSSION

We developed a novel approach to annotate EHR activities by augmenting raw audit logs with additional report detail and using action embeddings to learn the context and structure of EHR work. This approach enabled the identification of activities typically used in stereotyped specialized workflows (ie, mobile client use, preoperative assessment). In addition, we identified 2 distinct EHR information-gathering strategies used during chart biopsy activities, which could reflect setting-specific workflows (eg, inpatient vs outpatient) or individual-level variation in EHR use. Our approach also learned that other activities (such as placing orders, writing notes, and inbox messages) tended to occur across diverse contexts, potentially reflecting the fragmented nature of clinician work.²³^,²⁴

Our embeddings approach provides insights—complementary to clinician expert audit log annotations—into the context in which individual EHR actions occur. By situating specific actions within the broader context of their cooccurring activities, our method provides opportunities to generate meaningful hypotheses regarding clinical work activities, clinical workflows, and user interactions with the EHR. For example, the identification of distinct workflows for the same purpose creates opportunities for studying workflow efficiency and its contribution to outcomes such as burnout. This work also represents an important methodological advance in raw audit log research. Because our method treats audit log activities as tokens, it can be applied across any EHR raw audit log source, regardless of vendor or institution. Recent research has utilized raw audit logs to measure physician workload,³^,¹⁴^,²⁵^,²⁶ workflow,27–32 teamwork,³³ cognitive burden,¹⁹^,³⁴ and burnout,¹⁴^,¹⁵ illustrating the considerable potential of raw audit log data to capture physician behaviors at the macroscale. Our work differs from previous unsupervised methods for audit log analysis28–30 (1) by introducing audit log augmentation, without which our results would have been less insightful (Supplementary Figure S2) and (2) by focusing on individual actions, their context of use, and how they relate to each other with the proposed similarity score and global groupings, which fundamentally differs from the use of unsupervised methods for task identification as described previously.28–30

Our embeddings approach also enables the use of raw audit log data in machine learning research, because it transforms audit log data into numerical vectors on which computation can be performed. Because clinicians spend the majority of their time at work using the EHR,3–5 raw audit log activities may reflect not only clinician workload but also behavioral characteristics such as distractibility, which have the potential to be used to predict physician behaviors, errors, and outcomes. Traditional machine learning requires painstaking manual engineering of features to represent these complex concepts; in contrast, applying embeddings to raw audit log data enables the use of deep learning, which can directly learn these representations from the raw data. Recently, we demonstrated that combining embeddings with a hierarchical deep learning model can better predict physician burnout from raw audit log data compared to traditional feature engineering approaches.¹⁵^,³⁵

This work has several limitations. This was a single-center study on a small sample of intern physicians working across diverse clinical settings and specialties. Although our work has a strong theoretical foundation from NLP, further research is needed to establish the applicability of embedding representations across other EHR vendor platforms, clinician types, and work settings. For example, cluster analysis to identify workflows may not be as effective when the clinician population is more homogenous. In addition, the thresholds for identification of actions with high and low average neighbor similarity were empirically chosen based on the distribution of this metric in our data; different thresholds may apply for other datasets. We did not intend for our trained embedding vectors and analysis to be taken as representative of global physician EHR use; rather we presented a methodology for analyzing raw EHR audit logs and demonstrated its usefulness in 1 sample dataset as an example. We encourage the application of our methodology for deriving insights into EHR use patterns of other clinician groups or specific settings in future research.

Supplementary Material

ocac239_Supplementary_Data

Click here for additional data file.^{(327.9KB, docx)}

Contributor Information

Sunny S Lou, Department of Anesthesiology, School of Medicine, Washington University in St Louis, St Louis, Missouri, USA; Institute for Informatics, School of Medicine, Washington University in St Louis, St Louis, Missouri, USA.

Hanyang Liu, Department of Computer Science and Engineering, McKelvey School of Engineering, Washington University in St Louis, St Louis, Missouri, USA.

Derek Harford, Department of Anesthesiology, School of Medicine, Washington University in St Louis, St Louis, Missouri, USA.

Chenyang Lu, Department of Computer Science and Engineering, McKelvey School of Engineering, Washington University in St Louis, St Louis, Missouri, USA.

Thomas Kannampallil, Department of Anesthesiology, School of Medicine, Washington University in St Louis, St Louis, Missouri, USA; Institute for Informatics, School of Medicine, Washington University in St Louis, St Louis, Missouri, USA; Department of Computer Science and Engineering, McKelvey School of Engineering, Washington University in St Louis, St Louis, Missouri, USA.

FUNDING

This study was funded by the Washington University/BJC Healthcare Big Ideas Healthcare Innovation Competition. SSL was also supported by NIH 5T32GM108539-07.

AUTHOR CONTRIBUTIONS

SSL, HL, and TK contributed to the concept and design of the study. SSL, DH, and TK collected the data. SSL and HL developed the methodology and analyzed the data. SSL and TK interpreted the results. CL and TK supervised the work. SSL drafted the initial article. All authors contributed to the editing and review of the article and provided approval for publication.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

CONFLICT OF INTEREST STATEMENT

None declared.

DATA AVAILABILITY

Computer code to extract detailed audit log data and to train and analyze audit log action embeddings is available at: https://github.com/sslou/publications/tree/main/2022_embeddings/. The data that support the findings of this study are available from the corresponding authors on reasonable request. The data are not publicly available due to information that could compromise the privacy of study subjects.

REFERENCES

1. Office of the National Coordinator for Health Information Technology. Office-Based Physician Electronic Health Record Adoption. Health IT Quick-Stat #50; 2019. dashboard.healthit.gov/quickstats/pages/physician-ehr-adoption-trends.php. Accessed January 23, 2020.
2. Office of the National Coordinator for Health Information Technology. Non-Federal Acute Care Hospital Electronic Health Record Adoption. Health IT Quick-Stat #47; 2019. /quickstats/pages/FIG-Hospital-EHR-Adoption.php. Accessed January 23, 2020.
3. Ouyang D, Chen JH, Hom J, Chi J.. Internal medicine resident computer usage: an electronic audit of an inpatient service. JAMA Intern Med 2016; 176 (2): 252–4. [DOI] [PMC free article] [PubMed] [Google Scholar]
4. Tai-Seale M, Olson CW, Li J, et al. Electronic health record logs indicate that physicians split time evenly between seeing patients and desktop medicine. Health Aff (Millwood) 2017; 36 (4): 655–62. [DOI] [PMC free article] [PubMed] [Google Scholar]
5. Arndt BG, Beasley JW, Watkinson MD, et al. Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations. Ann Fam Med 2017; 15 (5): 419–26. [DOI] [PMC free article] [PubMed] [Google Scholar]
6. Rossetti S, Rosenbloom ST.. 25 by 5 Initiative to Reduce Documentation Burden on U.S. Clinicians by 75% by 2025 Summary Report. American Medical Informatics Association; 2021. https://brand.amia.org/m/dbde97860f393e1. Accessed July 11, 2022.
7. Nguyen OT, Jenkins NJ, Khanna N, et al. A systematic review of contributing factors of and solutions to electronic health record-related impacts on physician well-being. J Am Med Inform Assoc 2021; 28 (5): 974–84. [DOI] [PMC free article] [PubMed] [Google Scholar]
8. Moy AJ, Schwartz JM, Chen R, et al. Measurement of clinical documentation burden among physicians and nurses using electronic health records: a scoping review. J Am Med Inform Assoc 2021; 28 (5): 998–1008. [DOI] [PMC free article] [PubMed] [Google Scholar]
9. Adler-Milstein J, Adelman JS, Tai-Seale M, Patel VL, Dymek C.. EHR audit logs: a new goldmine for health services research? J Biomed Inform 2020; 101: 103343. [DOI] [PubMed] [Google Scholar]
10. Zheng K, Ratwani RM, Adler-Milstein J.. Studying workflow and workarounds in electronic health record-supported work to improve health system performance. Ann Intern Med 2020; 172 (11 Suppl): S116–S122. [DOI] [PMC free article] [PubMed] [Google Scholar]
11. Rule A, Chiang MF, Hribar MR.. Using electronic health record audit logs to study clinical activity: a systematic review of aims, measures, and methods. J Am Med Inform Assoc 2020; 27 (3): 480–90. [DOI] [PMC free article] [PubMed] [Google Scholar]
12. Sinsky CA, Rule A, Cohen G, et al. Metrics for assessing physician activity using electronic health record log data. J Am Med Inform Assoc 2020; 27 (4): 639–43. [DOI] [PMC free article] [PubMed] [Google Scholar]
13. Hron JD, Lourie E.. Have you got the time? Challenges using vendor electronic health record metrics of provider efficiency. J Am Med Inform Assoc 2020; 27 (4): 644–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
14. Lou SS, Lew D, Harford D, et al. Temporal associations between EHR-derived workload, burnout, and errors: a prospective cohort study. J Gen Intern Med 2022; 37 (9): 2165–72. [DOI] [PMC free article] [PubMed] [Google Scholar]
15. Lou SS, Liu H, Warner BC, Harford D, Lu C, Kannampallil T.. Predicting physician burnout using clinical activity logs: model performance and lessons learned. J Biomed Inform 2022; 127: 104015. [DOI] [PMC free article] [PubMed] [Google Scholar]
16. Mikolov T, Chen K, Corrado G, Dean J. Efficient estimation of word representations in vector space. In: workshop proceedings, international conference on learning representations; 2013; Scottsdale, AZ. doi: 10.48550/arXiv.1301.3781. [DOI]
17. Harris ZS. Distributional structure. WORD 1954; 10 (2–3): 146–62. doi: 10.1080/00437956.1954.11659520 [DOI] [Google Scholar]
18. Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J.. Distributed representations of words and phrases and their compositionality. In: proceedings of the 25th International Conference on Neural Information Processing Systems. Vol 2. Association for Computing Machinery; 2013: 3111–9. doi: 10.48550/arXiv.1310.4546. [DOI] [Google Scholar]
19. Lou SS, Kim S, Harford D, et al. Effect of clinician attention switching on workload and wrong-patient errors. Br J Anaesth 2022. doi: 10.1016/j.bja.2022.04.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
20. Sergienya I, Schütze H. Learning better embeddings for rare words using distributional representations. In: proceedings of the 2015 conference on empirical methods in natural language processing. Association for Computational Linguistics; 2015: 280–5; Lisbon, Portugal. doi: 10.18653/v1/D15-1033 [DOI]
21. Mikolov T, Tau YW, Zweig G. Linguistic regularities in continuous space word representations. In: proceedings of the 2013 conference of the North American chapter of the Association for Computational Linguistics: human language technologies. Association for Computational Linguistics; 2013: 746–51; Atlanta, GA. Accessed May 18, 2022. https://aclanthology.org/N13-1090
22. L van der M, Hinton G.. Visualizing data using t-SNE. J Mach Learn Res 2008; 9 (86): 2579–605. [Google Scholar]
23. Westbrook JI, Ampt A, Kearney L, Rob MI.. All in a day’s work: an observational study to quantify how and with whom doctors on hospital wards spend their time. Med J Aust 2008; 188 (9): 506–9. [DOI] [PubMed] [Google Scholar]
24. Ratanawongsa N, Matta GY, Lyles CR, et al. Multitasking and silent electronic health record use in ambulatory visits. JAMA Intern Med 2017; 177 (9): 1382–5. [DOI] [PMC free article] [PubMed] [Google Scholar]
25. Cox ML, Farjat AE, Risoli TJ, et al. Documenting or operating: where is time spent in general surgery residency? J Surg Educ 2018; 75 (6): e97–e106. [DOI] [PMC free article] [PubMed] [Google Scholar]
26. Dziorny AC, Orenstein EW, Lindell RB, Hames NA, Washington N, Desai B.. Automatic detection of front-line clinician hospital shifts: a novel use of electronic health record timestamp data. Appl Clin Inform 2019; 10 (1): 28–37. [DOI] [PMC free article] [PubMed] [Google Scholar]
27. Zheng K, Padman R, Johnson MP, Diamond HS.. An interface-driven analysis of user interactions with an electronic health records system. J Am Med Inform Assoc 2009; 16 (2): 228–37. [DOI] [PMC free article] [PubMed] [Google Scholar]
28. Perros I, Yan X, Jones JB, Sun J, Stewart WF.. Using the PARAFAC2 tensor factorization on EHR audit data to understand PCP desktop work. J Biomed Inform 2020; 101: 103312. [DOI] [PubMed] [Google Scholar]
29. Jones B, Zhang X, Malin BA, Chen Y.. Learning tasks of pediatric providers from electronic health record audit logs. AMIA Annu Symp Proc 2021; 2020: 612–8. [PMC free article] [PubMed] [Google Scholar]
30. Chen B, Alrifai W, Gao C, et al. Mining tasks and task characteristics from electronic health record audit logs with unsupervised machine learning. J Am Med Inform Assoc 2021; 28 (6): 1168–77. [DOI] [PMC free article] [PubMed] [Google Scholar]
31. Gong JJ, Soleimani H, Murray SG, Adler-Milstein J.. Characterizing styles of clinical note production and relationship to clinical work hours among first-year residents. J Am Med Inform Assoc 2021; 29 (1): 120–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
32. Noshad M, Rose CC, Chen JH.. Signal from the noise: a mixed graphical and quantitative process mining approach to evaluate care pathways applied to emergency stroke care. J Biomed Inform 2022; 127: 104004. [DOI] [PMC free article] [PubMed] [Google Scholar]
33. Li P, Chen B, Rhodes E, et al. Measuring collaboration through concurrent electronic health record usage: network analysis study. JMIR Med Inform 2021; 9 (9): e28998. [DOI] [PMC free article] [PubMed] [Google Scholar]
34. Lieu TA, Warton EM, East JA, et al. Evaluation of attention switching and duration of electronic inbox work among primary care physicians. JAMA Netw Open 2021; 4 (1): e2031856. [DOI] [PMC free article] [PubMed] [Google Scholar]
35. Liu H, Lou SS, Warner BC, Harford DR, Kannampallil T, Lu C. HiPAL: a deep framework for physician burnout prediction using activity logs in electronic health records. In: proceedings of the 28th ACM SIGKDD international conference on knowledge discovery & data mining (KDD ’22). Association for Computing Machinery; 2022; Washington, DC. doi: 10.1145/3534678.3539056 [DOI]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

ocac239_Supplementary_Data

Click here for additional data file.^{(327.9KB, docx)}

Data Availability Statement

[ocac239-B1] 1. Office of the National Coordinator for Health Information Technology. Office-Based Physician Electronic Health Record Adoption. Health IT Quick-Stat #50; 2019. dashboard.healthit.gov/quickstats/pages/physician-ehr-adoption-trends.php. Accessed January 23, 2020.

[ocac239-B2] 2. Office of the National Coordinator for Health Information Technology. Non-Federal Acute Care Hospital Electronic Health Record Adoption. Health IT Quick-Stat #47; 2019. /quickstats/pages/FIG-Hospital-EHR-Adoption.php. Accessed January 23, 2020.

[ocac239-B3] 3. Ouyang D, Chen JH, Hom J, Chi J.. Internal medicine resident computer usage: an electronic audit of an inpatient service. JAMA Intern Med 2016; 176 (2): 252–4. [DOI] [PMC free article] [PubMed] [Google Scholar]

[ocac239-B4] 4. Tai-Seale M, Olson CW, Li J, et al. Electronic health record logs indicate that physicians split time evenly between seeing patients and desktop medicine. Health Aff (Millwood) 2017; 36 (4): 655–62. [DOI] [PMC free article] [PubMed] [Google Scholar]

[ocac239-B5] 5. Arndt BG, Beasley JW, Watkinson MD, et al. Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations. Ann Fam Med 2017; 15 (5): 419–26. [DOI] [PMC free article] [PubMed] [Google Scholar]

[ocac239-B6] 6. Rossetti S, Rosenbloom ST.. 25 by 5 Initiative to Reduce Documentation Burden on U.S. Clinicians by 75% by 2025 Summary Report. American Medical Informatics Association; 2021. https://brand.amia.org/m/dbde97860f393e1. Accessed July 11, 2022.

[ocac239-B7] 7. Nguyen OT, Jenkins NJ, Khanna N, et al. A systematic review of contributing factors of and solutions to electronic health record-related impacts on physician well-being. J Am Med Inform Assoc 2021; 28 (5): 974–84. [DOI] [PMC free article] [PubMed] [Google Scholar]

[ocac239-B8] 8. Moy AJ, Schwartz JM, Chen R, et al. Measurement of clinical documentation burden among physicians and nurses using electronic health records: a scoping review. J Am Med Inform Assoc 2021; 28 (5): 998–1008. [DOI] [PMC free article] [PubMed] [Google Scholar]

[ocac239-B9] 9. Adler-Milstein J, Adelman JS, Tai-Seale M, Patel VL, Dymek C.. EHR audit logs: a new goldmine for health services research? J Biomed Inform 2020; 101: 103343. [DOI] [PubMed] [Google Scholar]

[ocac239-B10] 10. Zheng K, Ratwani RM, Adler-Milstein J.. Studying workflow and workarounds in electronic health record-supported work to improve health system performance. Ann Intern Med 2020; 172 (11 Suppl): S116–S122. [DOI] [PMC free article] [PubMed] [Google Scholar]

[ocac239-B11] 11. Rule A, Chiang MF, Hribar MR.. Using electronic health record audit logs to study clinical activity: a systematic review of aims, measures, and methods. J Am Med Inform Assoc 2020; 27 (3): 480–90. [DOI] [PMC free article] [PubMed] [Google Scholar]

[ocac239-B12] 12. Sinsky CA, Rule A, Cohen G, et al. Metrics for assessing physician activity using electronic health record log data. J Am Med Inform Assoc 2020; 27 (4): 639–43. [DOI] [PMC free article] [PubMed] [Google Scholar]

[ocac239-B13] 13. Hron JD, Lourie E.. Have you got the time? Challenges using vendor electronic health record metrics of provider efficiency. J Am Med Inform Assoc 2020; 27 (4): 644–6. [DOI] [PMC free article] [PubMed] [Google Scholar]

[ocac239-B14] 14. Lou SS, Lew D, Harford D, et al. Temporal associations between EHR-derived workload, burnout, and errors: a prospective cohort study. J Gen Intern Med 2022; 37 (9): 2165–72. [DOI] [PMC free article] [PubMed] [Google Scholar]

[ocac239-B15] 15. Lou SS, Liu H, Warner BC, Harford D, Lu C, Kannampallil T.. Predicting physician burnout using clinical activity logs: model performance and lessons learned. J Biomed Inform 2022; 127: 104015. [DOI] [PMC free article] [PubMed] [Google Scholar]

[ocac239-B16] 16. Mikolov T, Chen K, Corrado G, Dean J. Efficient estimation of word representations in vector space. In: workshop proceedings, international conference on learning representations; 2013; Scottsdale, AZ. doi: 10.48550/arXiv.1301.3781. [DOI]

[ocac239-B17] 17. Harris ZS. Distributional structure. WORD 1954; 10 (2–3): 146–62. doi: 10.1080/00437956.1954.11659520 [DOI] [Google Scholar]

[ocac239-B18] 18. Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J.. Distributed representations of words and phrases and their compositionality. In: proceedings of the 25th International Conference on Neural Information Processing Systems. Vol 2. Association for Computing Machinery; 2013: 3111–9. doi: 10.48550/arXiv.1310.4546. [DOI] [Google Scholar]

[ocac239-B19] 19. Lou SS, Kim S, Harford D, et al. Effect of clinician attention switching on workload and wrong-patient errors. Br J Anaesth 2022. doi: 10.1016/j.bja.2022.04.012. [DOI] [PMC free article] [PubMed] [Google Scholar]

[ocac239-B20] 20. Sergienya I, Schütze H. Learning better embeddings for rare words using distributional representations. In: proceedings of the 2015 conference on empirical methods in natural language processing. Association for Computational Linguistics; 2015: 280–5; Lisbon, Portugal. doi: 10.18653/v1/D15-1033 [DOI]

[ocac239-B21] 21. Mikolov T, Tau YW, Zweig G. Linguistic regularities in continuous space word representations. In: proceedings of the 2013 conference of the North American chapter of the Association for Computational Linguistics: human language technologies. Association for Computational Linguistics; 2013: 746–51; Atlanta, GA. Accessed May 18, 2022. https://aclanthology.org/N13-1090

[ocac239-B22] 22. L van der M, Hinton G.. Visualizing data using t-SNE. J Mach Learn Res 2008; 9 (86): 2579–605. [Google Scholar]

[ocac239-B23] 23. Westbrook JI, Ampt A, Kearney L, Rob MI.. All in a day’s work: an observational study to quantify how and with whom doctors on hospital wards spend their time. Med J Aust 2008; 188 (9): 506–9. [DOI] [PubMed] [Google Scholar]

[ocac239-B24] 24. Ratanawongsa N, Matta GY, Lyles CR, et al. Multitasking and silent electronic health record use in ambulatory visits. JAMA Intern Med 2017; 177 (9): 1382–5. [DOI] [PMC free article] [PubMed] [Google Scholar]

[ocac239-B25] 25. Cox ML, Farjat AE, Risoli TJ, et al. Documenting or operating: where is time spent in general surgery residency? J Surg Educ 2018; 75 (6): e97–e106. [DOI] [PMC free article] [PubMed] [Google Scholar]

[ocac239-B26] 26. Dziorny AC, Orenstein EW, Lindell RB, Hames NA, Washington N, Desai B.. Automatic detection of front-line clinician hospital shifts: a novel use of electronic health record timestamp data. Appl Clin Inform 2019; 10 (1): 28–37. [DOI] [PMC free article] [PubMed] [Google Scholar]

[ocac239-B27] 27. Zheng K, Padman R, Johnson MP, Diamond HS.. An interface-driven analysis of user interactions with an electronic health records system. J Am Med Inform Assoc 2009; 16 (2): 228–37. [DOI] [PMC free article] [PubMed] [Google Scholar]

[ocac239-B28] 28. Perros I, Yan X, Jones JB, Sun J, Stewart WF.. Using the PARAFAC2 tensor factorization on EHR audit data to understand PCP desktop work. J Biomed Inform 2020; 101: 103312. [DOI] [PubMed] [Google Scholar]

[ocac239-B29] 29. Jones B, Zhang X, Malin BA, Chen Y.. Learning tasks of pediatric providers from electronic health record audit logs. AMIA Annu Symp Proc 2021; 2020: 612–8. [PMC free article] [PubMed] [Google Scholar]

[ocac239-B30] 30. Chen B, Alrifai W, Gao C, et al. Mining tasks and task characteristics from electronic health record audit logs with unsupervised machine learning. J Am Med Inform Assoc 2021; 28 (6): 1168–77. [DOI] [PMC free article] [PubMed] [Google Scholar]

[ocac239-B31] 31. Gong JJ, Soleimani H, Murray SG, Adler-Milstein J.. Characterizing styles of clinical note production and relationship to clinical work hours among first-year residents. J Am Med Inform Assoc 2021; 29 (1): 120–7. [DOI] [PMC free article] [PubMed] [Google Scholar]

[ocac239-B32] 32. Noshad M, Rose CC, Chen JH.. Signal from the noise: a mixed graphical and quantitative process mining approach to evaluate care pathways applied to emergency stroke care. J Biomed Inform 2022; 127: 104004. [DOI] [PMC free article] [PubMed] [Google Scholar]

[ocac239-B33] 33. Li P, Chen B, Rhodes E, et al. Measuring collaboration through concurrent electronic health record usage: network analysis study. JMIR Med Inform 2021; 9 (9): e28998. [DOI] [PMC free article] [PubMed] [Google Scholar]

[ocac239-B34] 34. Lieu TA, Warton EM, East JA, et al. Evaluation of attention switching and duration of electronic inbox work among primary care physicians. JAMA Netw Open 2021; 4 (1): e2031856. [DOI] [PMC free article] [PubMed] [Google Scholar]

[ocac239-B35] 35. Liu H, Lou SS, Warner BC, Harford DR, Kannampallil T, Lu C. HiPAL: a deep framework for physician burnout prediction using activity logs in electronic health records. In: proceedings of the 28th ACM SIGKDD international conference on knowledge discovery & data mining (KDD ’22). Association for Computing Machinery; 2022; Washington, DC. doi: 10.1145/3534678.3539056 [DOI]

PERMALINK

Characterizing the macrostructure of electronic health record work using raw audit logs: an unsupervised action embeddings approach

Sunny S Lou

Hanyang Liu

Derek Harford

Chenyang Lu

Thomas Kannampallil

Abstract

INTRODUCTION