Predicting Physician Burnout using Clinical Activity Logs: Model Performance and Lessons Learned

Abstract

Background:

Burnout is a significant public health concern affecting more than half of the healthcare workforce; however, passive screening tools to detect burnout are lacking. We investigated the ability of machine learning (ML) techniques to identify burnout using passively collected electronic health record (EHR)-based audit log data.

Method:

Physician trainees participated in a longitudinal study where they completed monthly burnout surveys and provided access to their EHR-based audit logs. Using the monthly burnout scores as the target outcome, we trained ML models using combinations of features derived from audit log data—aggregate measures of clinical workload, time series-based temporal measures of EHR use, and the baseline burnout score. Five ML models were constructed to predict burnout as a continuous score: penalized linear regression, support vector machine, neural network, random forest, and gradient boosting machine.

Results:

88 trainee physicians participated and completed 416 surveys; >10 million audit log actions were collected (Mean [Standard Deviation]=25,691 [14,331] actions per month, per physician). The workload feature set predicted burnout score with a mean absolute error (MAE) of 0.602 (95% Confidence Interval (CI), 0.412–0.826), and was able to predict burnout status with an average AUROC of 0.595 (95% CI 0.355–0.808) and average accuracy 0.567 (95% CI 0.393–0.742). The temporal feature set had a similar performance, with MAE 0.596 (95% CI 0.391–0.826), and AUROC 0.581 (95% CI 0.343–0.790). The addition of the baseline burnout score to the workload features improved the model performance to a mean AUROC of 0.829 (95% CI 0.607–0.996) and mean accuracy of 0.781 (95% CI 0.587–0.936); however, this performance was not meaningfully different than using the baseline burnout score alone.

Conclusions:

Current findings illustrate the complexities of predicting burnout exclusively based on clinical work activities as captured in the EHR, highlighting its multi-factorial and individualized nature. Future prediction studies of burnout should account for individual factors (e.g., resilience, physiological measurements such as sleep) and associated system-level factors (e.g., leadership).

Introduction

Clinical settings place considerable psychological, emotional, and physical demands on practitioners, contributing to stress and burnout [1–3]. Physician burnout is widespread, with national surveys suggesting that it impacts nearly half of both physicians-in-training [4–7] and practicing physicians [8–12]—a prevalence twice that of the general population of the United States [11, 12]. Burnout has serious consequences: at a personal level, it is associated with poor physical and emotional health and suicidal ideation [5, 13, 14], alcohol abuse [15], and all-cause mortality [16]. For patients, it has been associated with lower quality of care [17], lesser patient satisfaction [18, 19], and increased risk for medical error [7, 18, 20, 21]; at an organizational level, it has been associated with increased provider turnover [22, 23] and decreased clinical productivity, costing the healthcare system over four billion dollars a year [24]. As such, burnout is a considerable public health concern, and tools for monitoring and managing clinician burnout are desperately needed [2]. However, current approaches to burnout measurement rely exclusively on surveys [25], with no known frameworks to unobtrusively monitor or predict clinician burnout.

Burnout is primarily a work-related phenomenon [3], and workload has been strongly correlated with burnout [26]. Measurement of workload has traditionally relied on self-reports, journals and time and motion studies [27–29]. With the widespread use of electronic health records (EHR), nearly 50% of a physician’s time is spent on EHR interactions [30–32]. Advances in clinical informatics have allowed researchers to leverage EHR-based audit logs—trails of activities performed on an EHR—to unobtrusively track clinical work activities and associated workload with considerable success [33, 34]. Previous studies have successfully conducted measurements of various contributors of clinical workload using audit logs, including administrative burden [35], cognitive load [36], interruptions [37], task switching [38], interface navigation [39], out-of-office work [30], and clinical documentation [40]. However, studies assessing the direct relationships between workload and burnout are very limited and have been focused on specific clinical tasks [41–44]. For example, Adler-Milstein et al. found that after-hours time spent on the EHR and inbox message volume among primary care physicians was associated increased emotional exhaustion, one of the three constructs associated with burnout [44]. Similarly, Tai-Seale and colleagues found associations between clinical inbox workload and burnout [43].

The ability of EHR audit logs to effectively capture workload measurements and its potential associations with burnout offers opportunities to develop a digital phenotype for burnout, similar to those that have been reported in the mental health literature [45, 46]. For example, machine learning (ML) techniques have used unobtrusively-collected mobile and wearable data [47–50] to predict depression, anxiety, and schizophrenia relapse in a variety of populations. Although burnout is not considered a mental health disorder [51], we hypothesized that burnout may be associated with changes in workload and work habits, which can potentially be captured from EHR audit logs.

The application of ML approaches for identifying burnout in clinicians has received limited attention. Towards this end, the primary objective of this study was to develop a predictive model for physician burnout using workload and work activity measures captured from EHR audit logs. Such a model can potentially be used as a screening tool to monitor and identify physicians at risk for burnout and to provide timely assistance and resources to those in need.

Method

Participants and Study Design

Intern physicians in Internal Medicine, Anesthesiology and Pediatrics training programs (N=104) and resident physicians in Internal Medicine (N=106) at the Washington University School of Medicine, BJC HealthCare and St Louis Children’s Hospital were invited to participate through presentations and email solicitations. Participating physician trainees worked across both inpatient and outpatient settings during the study period. Participants were consented and enrolled between September and November 2020, and data were collected through April 2021.

This was a prospective longitudinal cohort study. During the study period of 6 months, participants completed monthly burnout surveys and consented to provide their EHR-based audit logs and related activities. Participants were compensated $10 for the first survey completion and $5 for each ensuing survey completions.

This study was approved by the institutional review board of Washington University (IRB# 202004260) and was part of a larger study investigating the relationship between clinical workload and trainee wellness [52]. This study is reported according to TRIPOD guidelines [53].

Data Collection

All consented participants completed a baseline survey that included socio-demographic characteristics (age, race, sex, marital status, number of dependents), clinical domain and experience. Participants completed a monthly survey using the Stanford Professional Fulfillment Index (PFI) at baseline and consecutively for the following 6 months. PFI is a 16-item survey that combines burnout and professional fulfillment [21]; the burnout component of PFI is based on workload exhaustion and interpersonal disengagement (depersonalization) and correlates with the commonly used Maslach Burnout Inventory (MBI) on the emotional exhaustion and depersonalization scales [21, 54].

Given the longitudinal, monthly data collection, PFI has the advantage of being able to capture burnout over shorter time intervals, with the questions assessing burnout over the “past two weeks.” The surveys were electronically delivered via a secure REDCap link to each participant, aligned with their monthly rotation schedules to capture the burnout associated with the previous month’s schedule.

Synchronized with the monthly rotation schedules and burnout surveys, we also collected the raw EHR-based audit logs for each participant for the corresponding month. As previously described, audit log files are maintained in most modern EHRs to track and monitor access to protected health information as mandated by the Health Insurance Portability and Accountability Act [33]. Using the audit logs of consented participants, for each month and participant, we captured all EHR access events (e.g., opening a note, viewing a lab result, signing an order), including information on the time stamp, patient, and type of action performed.

Primary Outcome

The primary outcome was the monthly burnout, as assessed by the PFI scale; with a range of values from 0–4 [21]. Burnout score was used a continuous variable in the presented analysis, and hence, regressor models were employed for analyses.

Feature Engineering from Audit Log Files

EHR-based activities for all recruited participants were retrieved from institutional databases for the study period (Epic Systems, Verona WI). We chose to use raw audit log files as opposed to vendor-derived metrics such as Epic Signal, which can change over time with system upgrades [55], and are currently not available for inpatient workflows. Using the audit log data, several features associated with trainee workload and work patterns were generated, without knowledge of the outcome (i.e., burnout). We formulated two types of features—those based on aggregated “workload” measures and those that incorporated the raw “temporal” measures associated with clinical work activities. An overview of the feature extraction process from the audit log files is shown in Supplemental Figure S1.

The “workload” measures feature set was chosen based on prior research that showed that clinical workload was associated with burnout [41–44], and hence might be used to predict burnout. The primary work responsibilities of trainees typically include review of patient data, note writing, order placement, and review of clinical inbox messages. Therefore, similar to methods that have been previously developed [31], we identified all of the audit log actions relating to patient data review, notes, orders, or reviewing inbox messages; time stamps for the audit log actions were used to compute a heuristic approximation of the time spent on each action, as the difference in time stamps between each consecutive action. Additionally, similar to prior literature, durations of no activity extending for greater than 5 minutes were categorized as “inactive” time periods [31] (see Supplemental Figure S1). This method of measuring time spent on EHR activities has previously been shown to be correlated with time-motion measurements [30, 56].

We computed the following prespecified measures that summarized EHR-based clinical workload: total time spent using the EHR; time spent using the EHR after-hours, measured as the time spent between 6pm and 6am (similar to [57]), patient load, measured as the number of patients seen per day; time spent on the clinical inbox; number of ordering sessions per patient per day; time spent writing notes per patient per day; and time spent on chart review, i.e., reviewing patient notes, flowsheets, or laboratory test results, per patient, per day (Table 1). Each measure was aggregated over the month preceding each survey completion. These measures have been previously described as potential indicators of EHR-based work activities [58, 59]. The “workload” feature set included all of the above-mentioned pre-specified clinical workload measures as continuous variables, as well as one-hot encoded categorical variables for participant gender and specialty (Anesthesiology, Pediatrics and Internal Medicine).

Table 1.

Description for how EHR workload measures were calculated. Computer code for reproducing these measurements is available at https://github.com/sslou/publications/tree/main/2021_burnout_lmm. The method is illustrated in Supplemental Figure S1.

Metric	Description
Total EHR time per month (hours)	Divide audit log actions in the 30-day window preceding survey completion into chunks separated by at least 5 minutes of inactivity. Sum the difference in timestamps between the first action and last action in each chunk.
After-hours EHR time (hours/month)	Same as above, except only considering audit log actions that occur between 6pm and 6am.
Patient load per day	Mean number of patients per day for which the user either placed an order or wrote a note.
Inbox time per month (hours)	Assign time to audit log actions as the time between each action and subsequent action, excluding when that difference is greater than 5 minutes. Sum the time spent on all audit log actions assigned to the Inbox category.
Ordering sessions per patient per day	Count of audit log actions where METRIC_NAME == “Order list changed”, normalized per patient per day (measured with “Patient load per day” above). “Order list changed” appears in the audit log whenever a clinician signs one or more orders, including those placed using order sets. Thus, this variable captures the number of ordering sessions where at least one order is placed, rather than the total number of individual orders placed.
Note time per patient per day (hours)	Sum of total time spent on each note, measured using Epic metadata recording the time spent with the note-editing pane open for each note, normalized per patient per day using the number of patients per day for which the user wrote a note.
Review time per patient per day (hours)	Same as Inbox time, except for all audit log actions categorized to Note Review, Chart Review, or Results Review, and normalized per patient per day (measured with “Patient load per day” above)

For the “temporal” feature set, our assumption was that burnout might be associated with the temporal pattern of work activities on the EHR, potentially relating to increased cognitive load and decreased efficiency, and hence could be used to predict burnout. To summarize the temporal pattern of work activities, we measured the difference in time stamps between consecutive activities in the raw audit log and computed summary statistics for this vector including mean, minimum, maximum, skewness, kurtosis, entropy, total energy, autocorrelation, and slope [60].

Similarly, prior research has highlighted the increased cognitive load associated in the pattern of switches between tasks [61]. Therefore, we grouped actions in the raw audit log by patient and computed a vector representing the time spent consecutively on each patient’s chart before either inactivity (time gap > 5 min) or switching to a second patient’s chart. We summarized the temporal pattern of patient switches by computing the statistical features of this vector using similar summary statistics, as described above. The “temporal” feature set included the time series features summarizing the pattern of individual audit log actions and patient chart switches.

As burnout measurements for the same individual may be correlated over time [52], we also created a “first survey score” feature set that consisted of a single feature: a participant’s first recorded burnout score. This feature mimics the use of our model prospectively once a baseline survey was provided to help “calibrate” the model to the inter-individual variability. When the “first survey score” feature set was used, burnout surveys and associated EHR features from the first month of the study were excluded from the analysis to prevent label leakage.

The three feature sets (workload, temporal, and first survey score) were used alone and in combination to assess their relative contributions to burnout predictive performance (see Table 2 for a summary). To facilitate model training, all values of the extracted features were normalized and scaled to values between 0 and 1 using only the distribution of the training data.

Table 2.

Various combinations of feature sets that were utilized for model development: workload-based, temporal, first survey score, and combination of first survey score and workload. Two combinations of these feature sets were also considered: workload+temporal and first survey score+workload.

Feature set	Description
Workload	Gender and clinical specialty (as categorical variables). All of the EHR-associated features detailed in Table 1 (as continuous variables): total EHR time, after-hours EHR time, patient load, inbox time, time spent on notes, chart review, and number of orders, per patient per day.
Temporal	Statistical features (mean, minimum, maximum, skewness, kurtosis, entropy, total energy, autocorrelation, and slope) that describe the temporal pattern of time gaps between consecutive access events recorded in the raw audit log, and of time gaps for switches between different patient charts.
First survey score	PFI-based burnout score in the first month of study participation (i.e., baseline burnout score).

Study participants also occasionally worked at patient care locations where audit log data was not routinely available (e.g., at the Veterans Affairs). Therefore, we excluded all time periods (for a participant’s survey month) with fewer than 3000 recorded audit log actions. Otherwise, there was no missing data.

Model Development and Training

The primary focus of our analysis was to develop models that could be translated for predicting burnout outcomes for “unseen” participants. Towards this end, participant identifiers were not included in the model, and each monthly burnout survey, associated workload and temporal features were treated as independent measurements. For an overview of model training and evaluation, see Figure 1.

Figure 1.

Data collection, model training and cross-validation approach.

As previously described, our primary model formulation was as a regression problem, where we first estimated the continuous burnout scores and then used the same burnout threshold to dichotomize the estimated burnout.

Considering the relatively small sample size (i.e., total number of surveys), ML models may not yield robust performance given different splits of training and test datasets. To get as close an estimation as possible to the true out-of-sample performance of each model on unseen individuals, we used a repeated, nested cross-validation approach, synthesizing the process of hyperparameter optimization and model evaluation (Figure 1); in this nested process, the inner k-fold cross-validation procedure for model hyperparameter optimization was nested inside the outer k-fold cross-validation procedure for model evaluation. We chose 10-folds for both outer and inner cross-validation. For each fold of the outer tenfold cross-validation, the whole dataset was split into 90% training set and 10% testing set, grouped by participant such that data for any individual participant only appeared in either the training or testing set. This participant-level grouping was performed because measurements from the same participant were correlated over time. Then, a machine learning model was trained on the training set, using the optimal set of hyperparameters selected by randomly searching from a predefined hyperparameter space to maximize the mean of area under the receiver operating characteristic (AUROC) over the ten folds of the inner cross-validation performed on the training set. This best-performing trained model was then evaluated on the corresponding testing set for this fold. We repeated the above process for each of the ten folds of the outer cross-validation. Finally, the whole nested cross-validation was repeated 20 times using different random seeds for the k-fold splitting (see Figure 1).

The following regressor models were constructed: penalized linear regression, support vector machine, multi-layer perceptron neural network, random forest, and gradient boosting machine. The following hyperparameters were tuned within nested cross-validation: for penalized linear regression, penalty and L1/L2 ratio; for support vector machine, the kernel and regularization parameter; for neural network, the activation function, solver, and regularization parameter; for random forest, the number of features considered for each split and evaluation criterion; for gradient boosting, the learning rate, maximum depth, child weight, regularization, and the number of boosting rounds. All models were implemented using Python 3.9.5 and sci-kit learn 0.24.2, with the exception of the gradient boosting machine, which was implemented using XGBoost 1.4.0 [62].

Model Evaluation

Overall model performance in predicting burnout score was evaluated using mean absolute error (MAE). Each models’ ability to estimate the dichotomous burnout status was assessed across the range of possible predicted burnout score thresholds using the area under the receiver operating characteristic curve (AUROC). Accuracy was also evaluated by dichotomizing predicted and observed burnout scores using the previously validated burnout score threshold of 1.33 [21].

Results

General Characteristics

75 of 104 intern physicians and 13 of 106 resident physicians participated and provided a total of 528 surveys, of which 416 were completed (see Figure 1). A median of 6 (IQR 4–6) surveys were completed per participant. EHR audit log data associated with 25 surveys were excluded due to insufficient audit log events (<3000 actions). A total of 10,045,218 audit log actions were recorded across the study, with a mean (S.D.) of 25,691 (14,331) actions recorded per month per participant.

The demographics and summarized monthly workload measures associated with each completed survey are shown in Table 3 (also see Supplementary Material Table S1, for monthly temporal measures). 54% of the surveys were from Internal Medicine, 24% from Pediatrics, and 22% from Anesthesiology. 43% of surveys were from males. Participants spent a median of 88.3 (Interquartile Range (IQR) 53.0–124.1) hours each month using the EHR, with a median of 10.3 (IQR 4.4–29.8) hours occurring after-hours (i.e., between 6pm and 6am). Median patient load was 6.0 (IQR 4.8–7.1). Per patient per day, participants spent a median of 0.51 (IQR 0.37–0.72) hours writing notes and 0.49 (IQR 0.39–0.60) hours reviewing patient data. The median burnout score across the collected surveys was 1.8 (IQR 1.6–2.2). Burnout (PFI score ≥ 1.33) was identified in 164 surveys (42%).

Table 3.

Demographic characteristics of the survey responses and workload-associated EHR measures. Count and % presented for categorical variables. Median (IQR) presented for continuous variables.

Variable	Burned Out N = 164	Not Burned Out N = 227	All Survey Months N = 391
Specialty – Medicine	88 (54%)	124 (55%)	212 (54%)
Pediatrics	40 (24%)	76 (33%)	116 (30%)
Anesthesiology	36 (22%)	27 (12%)	63 (16%)
Gender (= Male)	70 (43%)	100 (44%)	170 (43%)
Avg total EHR time per month (hours)	94.5 (62.5–129.9)	81.4 (48.3–115.7)	88.3 (53.0–124.1)
Avg after-hours EHR time per month (hours)	13.0 (6.1–36.0)	8.7 (3.9–21.7)	10.3 (4.4–29.8)
Patient load	6.2 (5.3–7.3)	5.8 (4.6–7.0)	6.0 (4.8–7.1)
Number of orders/patient/day	3.4 (2.6–4.4)	3.2 (2.5–4.0)	3.2 (2.6–4.1)
Note time/patient/day (hours)	0.53 (0.40–0.72)	0.51 (0.35–0.72)	0.51 (0.37–0.72)
Review time/patient/day (hours)	0.50 (0.38–0.62)	0.49 (0.40–0.57)	0.49 (0.39–0.60)
Inbox time per month (hours)	0.98 (0.43–2.19)	1.21 (0.34–2.94)	1.07 (0.37–2.61)
1st survey-month PFI burnout score	1.8 (1.7–2.1)	1.0 (0.5–1.1)	1.3 (0.9–1.8)
Overall PFI burnout score	1.8 (1.6–2.2)	0.8 (0.3–1.1)	1.2 (0.7–1.7)

Model Performance

Five machine learning models were constructed to predict burnout as a continuous score: penalized linear regression, support vector machine, multi-layer perceptron neural network, random forest, and gradient boosting machine (Table 4).

Table 4.

Predictive performance for each feature set. Five ML regressor models were fit for each feature set to predict burnout score on the PFI scale (which ranges 1–4): linear regression, support vector machine, random forest, gradient boosting tree, and multi-layer perceptron neural network. The results for the best-performing model for each feature group is shown, along with the resulting mean absolute error (MAE) in burnout score prediction. The predicted score was dichotomized throughout the range of possible thresholds to measure area under the receiver operating characteristic curve (AUROC) for burnout identification. Accuracy is reported using the score threshold of 1.33. 95% confidence intervals are shown for all metrics as determined by nested cross-validation grouped by participant.

Feature Set	Best Model	MAE	AUROC	Accuracy
Workload	Random Forest	0.602 (0.412, 0.826)	0.595 (0.355, 0.808)	0.567 (0.393, 0.742)
Temporal	Support Vector Machine	0.596 (0.391, 0.826)	0.581 (0.343, 0.790)	0.556 (0.318, 0.756)
Workload + Temporal	Gradient Boosting Machine	0.619 (0.438, 0.844)	0.583 (0.270, 0.831)	0.559 (0.386, 0.780)
First Survey Score	Neural Network	0.432 (0.304, 0.570)	0.819 (0.551, 0.999)	0.765 (0.547, 0.952)
First Survey Score + Workload	Neural Network	0.423 (0.293, 0.567)	0.829 (0.607, 0.996)	0.781 (0.587, 0.936)

The “workload” feature set predicted burnout score with an average MAE of 0.602 (95% confidence interval CI, 0.412–0.826), and was able to predict burnout status with an average AUROC of 0.595 (95% CI 0.355–0.808) and average accuracy 0.567 (95% CI 0.393–0.742). The “temporal” feature set, had similar performance, with MAE 0.596 (95% CI 0.391–0.826), and AUROC 0.581 (95% CI 0.343–0.790). There was no increase in performance by combining the “workload” and “temporal” feature sets.

Due to the poor discriminative performance of the regression models, we conducted several secondary analyses. First, we evaluated model performance to directly classify burnout as a binary outcome, dichotomized using the previously validated 1.33 score threshold for burnout [21]. Discriminative performance was similar to the regression analysis (Supplementary Material Table S2).

Next, we also evaluated if the addition of the first reported burnout score (i.e., baseline score, at the start of the study) to the model (with workload features) could improve performance. With the addition of this baseline burnout score, model performance improved to a mean AUROC of 0.829 (95% CI 0.607–0.996) and mean accuracy of 0.781 (95% CI 0.587–0.936) (Table 4). However, knowledge of the baseline burnout alone had similar discriminative performance [AUROC=0.819 (95% CI 0.551–0.999), accuracy=0.765 (95% CI 0.547–0.952)].

Discussion

Using a longitudinal study of physician trainees, our goal was to develop a digital phenotype for burnout based on previously established associations between EHR-derived workload and burnout [11, 63, 64]. Our primary hypothesis, based on considerable prior literature, was that the EHR-based clinical activity logs would provide a proxy for developing meaningful EHR-based workload phenotypes associated with burnout. To the best of our knowledge, this is the first study attempting to develop a screening tool for burnout using passively collected data.

Using raw EHR-based audit log data, we engineered a panel of features that captured the primary work responsibilities of physician trainees, including all of the features previously shown to be associated with burnout in statistical analysis [41, 43, 44, 65], as well as time-series features that captured users’ EHR activity patterns. We found that the workload feature set predicted burnout with modest discriminative ability (average AUROC of 0.595 and average accuracy 0.567). The temporal feature set had similar performance (average AUROC 0.581 and average accuracy 0.556), suggesting that the pattern of a clinician’s EHR usage does provide some insight into their wellness status.

However, despite our best efforts, including both classification and regression approaches using multiple ML modeling strategies with various feature sets, none of the EHR-derived features were able to meaningfully identify clinicians suffering from burnout; confidence intervals for all EHR-derived feature performance spanned below AUROC 0.5, indicating substantial risk for model performance worse than random on new (i.e., unseen) individuals. Although the disappointing performance of these models can be attributed to several factors, our study raises several questions regarding the implicit relationships regarding EHR-based workload measures and burnout, and the viability of using EHR-derived measures alone in such modeling efforts [25].

First, while burnout is primarily a work-related phenomenon, individual responses to workload are likely highly individualized [11, 66]. In a mixed effects statistical analysis of the same cohort (not reported here), we found statistically significant associations between the same proximal clinical workload elements described here (total time spent on the EHR, number of patients, time spent on chart review) and burnout [52, 67] (based on a linear mixed effects regression; results not reported here). However, the measured effect size was modest in comparison to the variation in burnout observed. In addition, we found that the intra-class correlation of the individual level effects was 0.65, suggesting that measurements within individuals were highly correlated, and that more variance in burnout score could be attributed to individual effects than all of the workload measures combined. In the current analysis, we replicated that finding; the model using an individual’s first burnout survey response to predict all subsequent responses performed better than any of the models that used EHR measures alone (Table 4). In addition, inter-individual variability explains the wide confidence intervals for model performance we observed, as our cross-validation strategy was grouped by participant.

The addition of workload measures to the first survey response only marginally improved predictive performance (Table 4). We also found that our workload measures were no better at predicting the work exhaustion subscale of the burnout score, nor the degree of self-reported physical exhaustion at work (see summary in Supplemental Table S3). Taken together, these results suggest that although workload does contribute to the burnout phenotype, resulting in modest predictive ability (AUROC=0.595) of our workload-based model and the previously described statistical associations [41, 43, 44, 65], it may be that an individual’s interpretation or experience of their workload that is more important. In other words, two individuals exposed to the exact same workload may have very different burnout experiences. Future efforts to identify burnout likely need to account for these intrinsic individual-level variables in order to be successful.

Burnout is also a multi-factorial problem influenced by personal, contextual, and organizational factors in addition to workload itself [3]. In the current analysis, the model formulation primarily relied on workload-related factors and basic socio-demographic characteristics of participants. As such, the models did not account for the contextual aspects of the clinician’s personal behaviors (e.g., sleep, physical activity, coping skills, resilience, expectations, mental health) or that of the clinical work environment (e.g., interactions with patients, collaborative work with other clinicians). Additionally, as highlighted by recent research, system-level factors such as program leadership and other organizational characteristics also play a significant role in burnout [68–70], but were not incorporated in the current analysis. As described in a recent National Academy of Medicine report [2], burnout is likely more driven by the system-level factors such as work environment and its associated demands than individual-level factors.

Recently, there has been considerable research regarding the use of unobtrusive personal sensing frameworks using mobile and wearable platforms on a number of psychological constructs, feelings, traits and human behaviors [46]. These studies have relied on a number of measurements including that of physical activity, speech, behavioral indicators, location, sleep and related measures, and have had reasonably successful predictive performance (ranging from 0.65 to 0.8) [46, 71].

Perhaps the addition of more contextual and individualized measurement of physical and physiological activities—exercise, sleep, momentary assessments of wellness—using such platforms can provide situated assessments of individual characteristics that may improve burnout prediction. However, a point of concern is the fact that, as opposed to mood-related disorders that have associated symptomatology, some of the burnout symptoms (e.g., lack of empathy or lack of a feeling of personal accomplishment) are harder to capture with direct, unobtrusive measurement. In addition, recent research has highlighted that the symptoms and manifestations of burnout are distinct from that of depression or anxiety [72]. Nevertheless, future research should consider the use of mobile and wearable technologies and ecological momentary assessments for micro-measurement for further contextualizing the individual intangibles associated with burnout. Additionally, measures of resilience, coping abilities, and other mental health and wellness can provide additional context for burnout prediction.

Finally, our modeling efforts may also have been affected by work patterns of the considered study sample of residents. Residents are unique as they perform the role of learners and healthcare providers, often with limited autonomy. Their role as learners involves rotations in various clinical settings based on their clinical specialty. Such transitions between settings often result in varying clinical workload; for example, a clinical rotation in an intensive care unit versus one in an outpatient setting involve different work activities, patient characteristics, and time spent in the unit. This leads to constant fluctuations in workload and burnout, often on a month-to-month basis for the same trainee, contributing to the challenges in model-based prediction efforts. In addition, trainee physicians may have evolving EHR work habits over time as they gain experience. Modeling efforts may have more success among physicians in practice who have a more consistent work schedules (and workload) over time.

This study has several limitations. This was a single academic medical center study with a small group of trainee physicians and as such the findings may not translate to other settings or physician groups. The participation rate in this study was high, but all participants did not complete all their surveys, potentially leading to response bias. Survey completion was incentivized to facilitate participation; however, we could not assess the impact of the incentives on participation or survey responses. The number of burnout measurement cases (N=391 surveys) was moderate; a higher number of measurements, with a larger participant population may improve predictive performance by reducing overfitting or allowing the use of deep learning techniques to summarize raw audit log data and associated temporal features rather than relying on manual feature engineering.

Conclusions

Burnout is a significant public health concern, with more than half of healthcare workers being affected; the problem has been exacerbated during the COVID-19 pandemic highlighting the considerable challenges ahead for this unique workforce. Given that much of the clinical care activities are documented on the EHR, and that prior research has shown associations between clinical workload and activities and burnout, our focus was on using ML approaches to develop a EHR workload-based digital phenotype associated with burnout. However, workload metrics alone provided poor discriminative performance, highlighting the complexities of the burnout phenotype, especially its multi-factorial, and highly-individualized nature. Future work, especially in the prediction of burnout, should account for individual and behavioral characteristics that may provide novel discriminating features for prediction.