Abstract
Objectives This article investigates the association between changes in electronic health record (EHR) use during the coronavirus disease 2019 (COVID-19) pandemic on the rate of burnout, stress, posttraumatic stress disorder (PTSD), depression, and anxiety among physician trainees (residents and fellows).
Methods A total of 222 (of 1,375, 16.2%) physician trainees from an academic medical center responded to a Web-based survey. We compared the physician trainees who reported that their EHR use increased versus those whose EHR use stayed the same or decreased on outcomes related to depression, anxiety, stress, PTSD, and burnout using univariable and multivariable models. We examined whether self-reported exposure to COVID-19 patients moderated these relationships.
Results Physician trainees who reported increased use of EHR had higher burnout (adjusted mean, 1.48 [95% confidence interval [CI] 1.24, 1.71] vs. 1.05 [95% CI 0.93, 1.17]; p = 0.001) and were more likely to exhibit symptoms of PTSD (adjusted mean = 15.09 [95% CI 9.12, 21.05] vs. 9.36 [95% CI 7.38, 11.28]; p = 0.035). Physician trainees reporting increased EHR use outside of work were more likely to experience depression (adjusted mean, 8.37 [95% CI 5.68, 11.05] vs. 5.50 [95% CI 4.28, 6.72]; p = 0.035). Among physician trainees with increased EHR use, those exposed to COVID-19 patients had significantly higher burnout (2.04, p < 0.001) and depression scores (14.13, p = 0.003).
Conclusion Increased EHR use was associated with higher burnout, depression, and PTSD outcomes among physician trainees. Although preliminary, these findings have implications for creating systemic changes to manage the wellness and well-being of trainees.
Keywords: electronic health records, COVID-19, depression, burnout, physician trainees
Background and Significance
The coronavirus disease 2019 (COVID-19) pandemic has markedly impacted the mental health and well-being of health care workers around the world. 1 2 In parallel, the pandemic has created considerable changes in the clinical practice environment, including that of clinical workflows (e.g., new telemedicine workflows), guidelines for care (e.g., changes in the evidence-base for care), and modes of interaction and navigation within the electronic health record (EHR) (e.g., virtual visits). This rapid transformation required physicians to adapt their work activities. Perhaps, one of the most prominent changes was the surge in telemedicine use across health care systems. 3 4 5 6 Despite important benefits, including convenience and improved patient access, emerging evidence highlights the challenges these changes had on physicians, including the loss of personal connections and workflow issues associated with on-demand care. 7 8
Recent research has highlighted the mental health distress among physician trainees affecting burnout, stress, and anxiety. 9 10 11 12 Trainees have reported concerns about their safety, safety of their patients, implications of their decisions on their family, 13 challenges of child care, and lost educational opportunities 10 14 as key stressors during the pandemic. This is especially important given that trainees are a unique group of physicians functioning both as learners and care providers, often with limited autonomy or independence.
Even prior to the pandemic, EHR use was a key contributor to increased clinical workload, leading to stress, burnout, and adverse mental health outcomes. 15 16 17 18 19 20 With the pandemic-induced changes these effects are likely more pronounced. Because of the high documentation burden placed on physician trainees, 21 22 23 24 they spend a considerable amount of time on the EHR, 25 26 27 increasing their workload and burden. As such, the twin, potentially deleterious, effects of patient care during the pandemic and changes in EHR-based workflow can have a considerable impact on physician trainees and have not been previously studied.
The purpose of this study is twofold. First, we sought to investigate the relationship between changes in EHR use during the pandemic and the mental health and well-being of physician trainees, assessed as burnout, depression, anxiety, stress, and posttraumatic stress disorder (PTSD) outcomes. Second, we examined the potential interaction of exposure to COVID-19 patients on the relationship between EHR use and mental health outcomes.
Method
Participants and Survey
Physician trainees ( n = 1,375), both residents and fellows, at Washington University School of Medicine received a Web-based survey on June 19, 2020, with a follow-up reminder a week later. A $50 gift card raffle was offered to survey respondents for participation. The survey was approved by the institutional review board of Washington University (IRB #202004021). Survey invitation emails described the purpose of the survey as “to understand the impact of the COVID-19 pandemic on the well-being of physician trainees” and that the survey was part of a longitudinal study assessing the wellness of physician trainees during the COVID-19 pandemic. 10
The survey included demographic questions about race, sex, marital status, training program, and clinical role. Participants were also asked whether they were caring for patients who tested positive for COVID-19 (Y/N). Survey questions related to the five outcomes included: burnout, as measured with the Stanford Professional Fulfillment Index (PFI) 28 ; PTSD, as measured by the Impact of Event Scale–Revised (IES-R) 29 ; and depression, anxiety, and stress, as measured by the Depression Anxiety Stress Scale (DASS-21). 30
The 16-item PFI was used to measure the level of burnout (workload exhaustion and interpersonal disengagement) and professional fulfillment during the past 2 weeks. The PFI burnout scale has previously been validated with the Maslach Burnout Inventory, and the fulfillment measure is correlated with quality of life scores. 28 31 32
The IES-R scale was used to measure the amount of distress respondents felt with respect to the current COVID-19 pandemic. Responses were on a 5-item Likert scale ranging from “Not at all” to “Extremely.” The scores range from 0 to 88 with a score of 24 or higher indicating clinical concern for PTSD, including at least partial PTSD or some of its symptoms. 33
The DASS-21 is a 21-item scale that has demonstrated strong validity and reliability in clinical and nonclinical samples 34 35 and has been used with physician trainees. 36 This scale was used to ascertain level of depression, anxiety, and stress. 30
Exposure
We considered two exposures related to EHR use: changes in overall EHR use and EHR use outside of work (with an example of “home” as a setting outside of work). Respondents were asked about changes in their EHR usage since the start of the pandemic (March 2020), indicating whether their “overall EHR use” and “EHR use outside of work hours” have been the same, increased, or decreased.
Statistical Analysis
Physician trainee responses to their self-reported changes in overall EHR use and EHR use outside of work questions were categorized into two groups: increasing and same/decreasing. Race was categorized as Caucasian, Asian, black, Hispanic or Latino, and Other/Prefer not to answer; sex was categorized as female or not female; and marital status was categorized as married or not married.
Descriptive statistics for changes in EHR use (both overall and outside of work) among increasing and same/decreasing groups, outcomes from the DASS-21 were categorized as normal or nonnormal as follows: depression (0–9 normal, 10 or greater not normal), anxiety (0–7 normal, 8 or greater not normal), and stress (0–14 normal, 15 or greater not normal). These cut-points were based on previously published literature on the DASS-21 scale. 37 PTSD was categorized as a binary variable based on the diagnostic cutoff recommended by the IES-R (0–24 no PTSD, 24 or greater indicates clinically meaningful PTSD symptomatology). 38 39
Burnout was determined from the average item score for the workload and depersonalization scales (score range 0–4), using a cut-point of 1.33 as described in Trockel et al, 28 where scores greater than or equal to 1.33 were considered as “burned out.”
For univariable and multivariable analyses, burnout was analyzed as a continuous outcome variable, and outcomes from the DASS-21 (depression, anxiety, and stress) and IES-R (PTSD) were analyzed as count variables. Associations between the EHR use and burnout were analyzed using linear least-squares regression analyses; depression, anxiety, and stress outcomes were analyzed using negative binomial regression analyses. Exposure effects were adjusted in multivariable models by including covariates with p -values < 0.10 in the univariable analyses. The final multivariable models investigated possible interactions between trainees exposed to COVID-19 positive patients and increasing or same/decreasing overall EHR use and EHR use outside of work. p -Values < 0.05 were considered significant. All analyses were conducted using Stata/SE version 16.0. 40
Results
There were 250 responses to the survey. After removing 28 duplicate or incomplete entries, there were a total of 222 completed surveys (16.2% response).
General Characteristics
Demographic characteristics are presented in Table 1 . There were equal numbers of males and females in the sample. Participants were predominantly residents (70.3%), within the first 3 years of their training (78.3%), Caucasian (64.0%), married (58.1%), and with no children (77.5%). Twenty-seven percent of the participants reported that they were exposed to COVID-19 patients. Since the start of the pandemic, 21% of trainees reported increased EHR usage, and 33% reported increased EHR use outside of work.
Table 1. Summary descriptive table of the considered variables in the survey for all trainees.
| Variable | Group | All trainees ( n = 222) |
Overall EHR use increasing ( n = 46) |
Overall EHR use same/decreasing ( n = 175) | EHR use outside of work increasing ( n = 73) |
EHR use outside of work same/decreasing ( n = 148) |
|---|---|---|---|---|---|---|
| Clinical role | Fellow | 66 (29.7%) | 17 (37.0%) | 48 (27.4%) | 21 (28.8%) | 44 (29.7%) |
| Resident | 156 (70.3%) | 29 (63.0%) | 127 (72.6%) | 52 (71.2%) | 104 (70.4%) | |
| Female | Yes | 111 (50.0%) | 25 (54.4%) | 86 (49.1%) | 40 (54.8%) | 70 (47.3%) |
| Race | Caucasian | 142 (64.0%) | 30 (65.2%) | 111 (63.8%) | 48 (65.8%) | 93 (63.3%) |
| Asian | 54 (24.3%) | 8 (17.4%) | 46 (26.4%) | 16 (21.9%) | 38 (25.9%) | |
| Black | 5 (2.3%) | 2 (4.4%) | 3 (1.7%) | 1 (1.4%) | 4 (2.7%) | |
| Hispanic or Latino | 6 (2.7%) | 2 (4.4%) | 4 (2.3%) | 2 (2.7%) | 4 (2.7%) | |
| Other/Unknown | 14 (6.3%) | 4 (8.7%) | 10 (5.7%) | 6 (8.2%) | 8 (5.4%) | |
| Married | Yes | 129 (58.1%) | 25 (54.4%) | 103 (58.9%) | 41 (56.2%) | 87 (58.8%) |
| Children at home | Yes | 50 (22.5%) | 9 (19.6%) | 40 (22.9%) | 13 (17.8%) | 37 (25.0%) |
| ≥ 4 years of program | Yes | 48 (21.7%) | 5 (10.9%) | 43 (24.7%) | 15 (20.6%) | 33 (22.5%) |
| Exposed to COVID-19 positive patients | Yes | 81 (26.5%) | 17 (37.0%) | 63 (36.0%) | 21 (28.8%) | 59 (39.9%) |
| Depression | Mild to extremely severe | 61 (27.5%) | 16 (34.8%) | 45 (25.7%) | 27 (37.0%) | 33 (22.3%) |
| Anxiety | Mild to extremely severe | 42 (18.9%) | 10 (21.7%) | 32 (18.3%) | 18 (24.7%) | 24 (16.2%) |
| Stress | Mild to extremely severe | 50 (22.5%) | 15 (32.6%) | 35 (20.0%) | 21 (28.8%) | 28 (18.9%) |
| Posttraumatic stress disorder (IES-R ≥ 24) | Yes | 20 (9.0%) | 14 (30.4%) | 22 (12.6%) | 17 (23.3%) | 19 (12.8%) |
| Professional fulfillment (PFI ≥ 3) | Yes | 60 (27.0%) | 11 (23.9%) | 48 (27.4%) | 22 (30.1%) | 38 (25.7%) |
| Burnout (mean workload, depersonalization PFI ≥ 1.33) | Yes | 84 (37.8%) | 24 (52.2%) | 59 (33.7%) | 35 (48.0%) | 48 (32.4%) |
Abbreviations: COVID-19, coronavirus disease 2019; EHR, electronic health record; IES-R, Impact of Event Scale–Revised; PFI, Professional Fulfillment Index.
The distribution of participants on each of the considered predictors and outcomes of both the EHR use and EHR use outside of work variables are shown in Table 1 .
Multivariable Analysis
Findings from the multivariable model indicated that physician trainees with increased overall EHR use were more likely to be burned out (1.48 [95% confidence interval [CI], 1.24–1.71] vs. 1.05 [95% CI, 0.93–1.17]; p = 0.001) (see Table 2 ; for covariates see Appendix A ).
Table 2. Univariable and multivariable model results.
| Variable | Group | Unadjusted mean burnout score (95% CI) | Univariable p -value | Adjusted mean burnout score (95% CI) | Multivariable p -value |
|---|---|---|---|---|---|
| Overall EHR use | Same/Decreasing | 1.05 (0.93, 1.18) | 0.002 | 1.05 (0.93, 1.17) | 0.001 |
| Increasing | 1.46 (1.22, 1.69) | 1.48 (1.24, 1.71) | |||
| EHR use outside of work | Same/Decreasing | 1.06 (0.93, 1.20) | 0.051 | 1.05 (0.92, 1.19) | 0.026 |
| Increasing | 1.30 (1.11, 1.48) | 1.32 (1.13, 1.50) | |||
| Variable | Group | Unadjusted mean depression score (95% CI) | Univariable p -value | Adjusted mean depression score (95% CI) | Multivariable p -value |
| Overall EHR use | Same/Decreasing | 5.81 (4.61, 7.01) | 0.063 | 5.83 (4.62, 7.05) | 0.079 |
| Increasing | 8.89 (5.36, 12.42) | 8.77 (5.25, 12.29) | |||
| EHR use outside of work | Same/Decreasing | 5.51 (4.29, 6.74) | 0.039 | 5.50 (4.28, 6.72) | 0.035 |
| Increasing | 8.32 (5.65, 10.98) | 8.37 (5.68, 11.05) | |||
| Variable | Group | Unadjusted mean PTSD score (95% CI) | Univariable p -value | Adjusted mean PTSD score (95% CI) | Multivariable p -value |
| Overall EHR use | Same/Decreasing | 9.26 (7.35, 11.18) | 0.025 | 9.33 (7.38, 11.28) | 0.035 |
| Increasing | 15.49 (9.32, 21.65) | 15.09 (9.12, 21.05) | |||
| EHR use outside of work | Same/Decreasing | 9.14 (7.06, 11.23) | 0.068 | 9.29 (7.15, 11.44) | 0.114 |
| Increasing | 13.23 (8.93, 17.52) | 12.81 (8.68, 16.94) |
Abbreviations: CI, confidence interval; EHR, electronic health record; PTSD, posttraumatic stress disorder.
Note: Burnout was analyzed using linear least-squares regression and PTSD and depression-related outcomes were analyzed using negative binomial regression. Unadjusted means correspond to means unadjusted for covariates. Adjusted means correspond to means from multivariable models adjusted for variables with p < 0.10 in the univariable analyses (see Appendix A and B ). Bold indicates p value <.10.
Appendix A. Univariate linear least-squares regression models for burnout and covariates.
| Variable | Group | Unadjusted mean (95% CI) | Univariable p -value |
|---|---|---|---|
| Clinical role | Fellow | 0.99 (0.80, 1.19) | 0.079 |
| Resident | 1.21 (1.08, 1.34) | ||
| Caucasian | No | 1.14 (0.95, 1.32) | 0.918 |
| Yes | 1.15 (1.01, 1.28) | ||
| Female | No | 1.11 (0.95, 1.26) | 0.526 |
| Yes | 1.18 (1.02, 1.33) | ||
| Children at home | No | 1.19 (1.06, 1.31) | 0.128 |
| Yes | 0.99 (0.75, 1.22) | ||
| Married | No | 1.21 (1.04, 1.38) | 0.285 |
| Yes | 1.09 (0.95, 1.24) | ||
| Year in program | − | 0.01 (−0.08, 0.10) | 0.767 |
| Exposed to COVID-19 positive patients | No | 1.05 (0.91, 1.18) | 0.021 |
| Yes | 1.31 (1.13, 1.49) |
Abbreviations: CI, confidence interval; COVID-19, coronavirus disease 2019.
Physician trainees reporting increased EHR use had increased symptoms of PTSD (15.09 [95% CI 9.12, 21.05] vs. 9.33 [95% CI 7.38, 11.28]; p = 0.035) (see Table 2 ; for covariates see Appendix B ). Those reporting increased EHR use outside of work had higher depression scores (8.37 [95% CI, 5.68, 11.05] vs. 5.50 [95% CI, 4.28, 6.72]; p = 0.035) (see Table 2 ; for covariates see Appendix C ). There were no significant differences between the EHR usage groups for stress or anxiety (see Appendix D and E ).
Appendix B. Univariate negative binomial regression models for PTSD and covariates.
| Variable | Group | Unadjusted mean (95% CI) | Univariable p -value |
|---|---|---|---|
| Clinical role | Fellow | 1.00 (7.26, 14.74) | 0.756 |
| Resident | 10.31 (8.01, 12.62) | ||
| Caucasian | No | 11.39 (7.88, 14.90) | 0.514 |
| Yes | 10.01 (7.66, 12.36) | ||
| Female | No | 8.65 (6.40, 10.91) | 0.054 |
| Yes | 12.45 (9.17, 15.73) | ||
| Children at home | No | 10.62 (8.36, 12.87) | 0.855 |
| Yes | 10.18 (6.19, 14.18) | ||
| Married | No | 10.76 (7.68, 13.84) | 0.836 |
| Yes | 10.34 (6.69, 12.89) | ||
| Year in program | − | 0.15 (−0.01, 0.30) | 0.064 |
| Exposed to COVID-19 positive patients | No | 9.94 (7.60, 12.28) | 0.460 |
| Yes | 11.50 (7.98, 15.02) |
Abbreviations: CI, confidence interval; COVID-19, coronavirus disease 2019; PTSD, posttraumatic stress disorder.
Appendix C. Univariate negative binomial regression models for depression and covariates.
| Variable | Group | Unadjusted mean (95% CI) | Univariable p -value |
|---|---|---|---|
| Clinical role | Fellow | 6.49 (4.31, 8.68) | 0.963 |
| Resident | 6.43 (5.02, 7.84) | ||
| Caucasian | No | 6.60 (4.60, 8.60) | 0.850 |
| Yes | 6.36 (4.89, 7.83) | ||
| Female | No | 5.78 (4.28, 7.28) | 0.265 |
| Yes | 7.12 (5.28, 8.96) | ||
| Children at home | No | 6.92 (5.49, 8.35) | 0.107 |
| Yes | 4.82 (2.94, 6.70) | ||
| Married | No | 6.86 (4.91, 8.80) | 0.571 |
| Yes | 6.16 (4.67, 7.64) | ||
| Year in program | 0.02 (−0.13, 0.17) | 0.805 | |
| Exposed to COVID-19 positive patients | No | 6.10 (4.69, 7.51) | 0.456 |
| Yes | 7.05 (4.92, 9.18) |
Abbreviations: CI, confidence interval; COVID-19, coronavirus disease 2019.
Appendix D. Univariate negative binomial regression models for stress and covariates.
| Variable | Group | Unadjusted mean (95% CI) | Univariable p -value |
|---|---|---|---|
| Overall EHR use | Same/Decreasing | 8.16 (6.71, 9.61) | 0.134 |
| Increasing | 10.98 (7.19, 14.76) | ||
| My EHR use outside of work | Same/Decreasing | 8.11 (6.55, 9.67) | 0.237 |
| Increasing | 9.94 (7.18, 12.70) | ||
| Clinical role | Fellow | 9.97 (7.10, 12.84) | 0.276 |
| Resident | 8.23 (6.68, 9.78) | ||
| Caucasian | No | 8.75 (6.46, 11.04) | 0.999 |
| Yes | 8.75 (7.01, 10.48) | ||
| Female | No | 7.83 (6.08, 9.59) | 0.194 |
| Yes | 9.65 (7.52, 11.79) | ||
| Children at home | No | 8.62 (7.08, 10.17) | 0.745 |
| Yes | 9.18 (6.12, 12.25) | ||
| Married | No | 8.22 (6.21, 10.22) | 0.517 |
| Yes | 9.13 (7.24, 11.02) | ||
| Year in program | 0.10 (−0.03, 0.26) | 0.137 | |
| Exposed to COVID-19 positive patients | No | 8.30 (6.65, 9.95) | 0.411 |
| Yes | 9.52 (7.05, 12.00) |
Abbreviations: CI, confidence interval; COVID-19, coronavirus disease 2019; EHR, electronic health record.
Appendix E. Univariate negative binomial regression models for anxiety and covariates.
| Variable | Group | Unadjusted mean (95% CI) | Univariable p -value |
|---|---|---|---|
| Overall EHR use | Same/Decreasing | 3.19 (2.25, 4.13) | 0.198 |
| Increasing | 4.84 (2.11, 7.58) | ||
| My EHR use outside of work | Same/Decreasing | 3.03 (2.05, 4.01) | 0.169 |
| Increasing | 4.48 (2.44, 6.51) | ||
| Clinical role | Fellow | 3.00 (1.57, 4.43) | 0.446 |
| Resident | 3.75 (2.57, 4.92) | ||
| Caucasian | No | 4.10 (2.31, 5.90) | 0.369 |
| Yes | 3.19 (2.15, 4.24) | ||
| Female | No | 2.92 (1.84, 4.00) | 0.193 |
| Yes | 4.13 (2.60, 5.66) | ||
| Children at home | No | 3.78 (2.66, 4.91) | 0.250 |
| Yes | 2.61 (1.16, 4.07) | ||
| Married | No | 4.07 (2.43, 5.70) | 0.328 |
| Yes | 3.12 (2.04, 4.20) | ||
| Year in program | 0.09 (−0.14, 0.32) | 0.438 | |
| Exposed to COVID-19 positive patients | No | 3.39 (2.27, 4.51) | 0.723 |
| Yes | 3.74 (2.11, 5.38) |
Abbreviations: CI, confidence interval; COVID-19, coronavirus disease 2019; EHR, electronic health record.
There was a significant interaction between self-reported caring for COVID-19 patients and changes in overall EHR use and burnout ( p = 0.003) and depression ( p = 0.032). Among the participants reporting increased EHR use, those caring for COVID-19 patients had a significantly higher mean burnout score (2.04, p < 0.001) than those whose EHR use increased but were not caring for COVID-19 patients. Burnout score for the group whose EHR use stayed the same or decreased was not significantly different regardless of their exposure to COVID-19 patients. Similarly, among physician trainees whose EHR use increased, those that were exposed to COVID-19 patients had a significantly higher mean depression score (14.13, p = 0.003) than those who were not caring for COVID-19 patients. The depression score for the group whose reported EHR use remained the same or decreased was not significantly different for those who were exposed compared with those who were not exposed to COVID-19 patients (see Fig. 1 ).
Fig. 1.
Exposure to COVID-19 positive patients moderates relationship between overall EHR use and levels of burnout (left) and depression (right).
Discussion
Based on a cross-sectional study of physician trainees, we found that self-reported increased EHR use, including EHR use outside of work settings, was associated with a higher likelihood of burnout, depression, and PTSD. An estimated 38% of physician trainees met the criteria for burnout, similar to results of a recent review of burnout among health care workers in general during COVID-19 (34.4%) 41 and slightly less than a study focusing specifically on surgery trainees during COVID-19 (39–45%). 9 Notably, the range of burnout among physician trainees prior to COVID-19 has varied widely with several studies reporting greater than 50%. 42 43 44 45 Results for depression (27.5%) were similar to those for health care workers in general (26.3%), however, less than the percentage of surgical trainees (30.0%). Anxiety was less common among trainees in the present study (18.9%) than health care workers (29.0%) and surgical residents (54%), though both of these studies utilized different scales. Among physician trainees reporting increased EHR use, the average levels of burnout and depression were significantly higher for those caring for COVID-19 patients, highlighting the additive impact on EHR-based changes and clinical care challenges. Although the high prevalence of burnout among trainees is well-known, 10 46 47 48 the prevalence of PTSD and depression associated with increased EHR use is potentially concerning. Of these, the contributory role of EHR use outside of work settings on depression highlights the effects of the clinical workload on mental health outcomes. As such, strategies for mitigating the effects of chronic burnout and related mental health issues are urgent among physician trainees to limit the loss of physician trainees from the workforce and to prevent a parallel pandemic, that leads to distress, loneliness, substance abuse, and other chronic illnesses. 49 50 51
Although we are not able to determine why some trainees reported increased EHR use, we speculate that it may be, in part, related to the increased need for virtual visits or additional tasks associated with the changes occurring during the COVID-19 pandemic. For example, telemedicine interactions have changed the mechanism of physician–patient interactions and are associated with increased administrative burden. 52 53 54 55 Telemedicine interactions are “touchless,” where physicians can no longer perform a physical exam or collect vitals. Instead, they rely on the patient's record for collecting information utilizing past medical history, notes, and video-based communication for their reasoning and clinical decision making. 56
Challenges of the video-based medium also increase the difficulties of communication, where patients can be less forthcoming, and the nuances that are available in a face-to-face conversation may be lost. 57 58 59 60 Such changes can also result in the need for additional documentation and for potentially additional clinical tests, all of which add to the clinical activities that need to be performed on the EHR. There are likely contributing factors to increased EHR use and burnout; although the effects of increased telehealth use may have been a contributor to burnout, we did not specifically investigate it in this study. As reported elsewhere, there are considerable advantages of telemedicine to both clinicians and patients. 61 62 Recent reports have highlighted the increased number of messages (phone, inbox) that physicians receive. 63 High volume of calls/messages may also have contributed to the increased EHR use, a factor that has been strongly associated with clinician burnout. 64
Increased EHR workload can also lead to worse mental health outcomes due to cognitive dissonance 65 —a mismatch between the trainees' need to perform mundane documentation tasks and their altruistic goals of patient care—where the twin challenges of additional cognitive work on the EHR and the external challenges and stress associated with the pandemic contributes to moral distress and burnout. Identifying and addressing the sources of such challenges can be an important first step in streamlining clinical care activities using the EHR. In addition, the additive effects of caring for patients during a pandemic with increased EHR-based work activities highlight the complex system-level factors that affect physician burnout and wellness. 51 66 Addressing these system-level factors would require reduction of nonphysician tasks, offloading physician tasks to nonphysician members of the team, improving clinical workflows, and streamlining interactions with EHRs. 67 68 Additional considerations of incorporating sociotechnical factors into the design of EHRs, and associated clinical work activities can also help in improving clinical work practices. 69
Limitations
This study has several limitations. First, this was a cross-sectional, single-center study. As such, the associations between the outcomes and exposure factors may not be causal and the findings should be interpreted with caution. It is possible that there was a response bias among the participants; participants who were distressed may not have completed the survey or may have been more likely to participate because of the relevance of the topic. The response rate for the survey was low, and is only marginally comparable to response rates in other surveys of clinical trainees 47 that were conducted prior to the pandemic. Survey response rates among physician trainees during the pandemic have been similar or lower. 9
Measurements related to EHR use and EHR use outside of work were based on self-report and may have been affected by recall bias. It is possible that “overall EHR use” was interpreted as a proxy for workload or the total number of hours worked. Physician trainees who worked more, may have potentially perceived themselves as using the EHR more. Further, these results represent a single time point during the pandemic, and although the questions were framed around EHR use and distress during the pandemic, it is difficult to ascertain how much of the change can be attributed to the pandemic rather than nonpandemic-related changes. Participants' prepandemic EHR practices and baseline mental health scores were unknown, and hence, it is difficult to ascertain the directionality of the relationship between these variables. It is possible that mental health issues are causing individuals to feel that their work is increasing.
Conclusion and Future Research
This study characterizes the unique relationship between clinical work activities that compound the stressors experienced by trainees. It was revealed that trainees have high rates of mental distress and those reporting more EHR use, particularly at home, had markedly higher rates of distress. Exposure to COVID-19 patients appears to have exacerbated these already high rates. These experiences are likely to be similar or more pronounced in other settings where the infection rates are higher.
Research relying on audit logs 70 71 72 73 or similar passive measurements to complement results of self-report measures can potentially be used to ascertain the true changes in EHR use during the pandemic. Future studies should also look at longitudinal relationships between EHR use, COVID-19 patient exposure, and mental health distress.
Clinical Relevance Statement
The outcomes of this study indicate that increased EHR use is associated with higher burnout, depression, and PTSD among physician trainees. Those trainees who are exposed to COVID-19 patients have significantly higher burnout and depression scores than trainees not caring for COVID-19 patients. The combination of these stressors and their concerning effects on mental health suggest the need for systemic changes to improve the well-being of trainees, particularly during this period of vulnerability. More effective methods to combat these issues include a reduction of nonphysician tasks with the help of nonphysician team members, improving clinical workflows, and streamlining EHR practices.
Multiple Choice Questions
-
Physician trainees report which of the following concerns as key stressors during the COVID-19 pandemic?
Their own safety
Safety of their patients
Implications of their decisions on their family
Challenges of childcare
Lost education opportunities
All of the above
Correct Answer: The correct answer is f. In research conducted during the COVID-19 pandemic, trainees report that their safety, safety of their patients, safety of their family, challenges of childcare, and lost education opportunities are key stressors.
-
What is an objective alternative to subjective self-reports for characterizing physician trainees' time spent on the EHR?
EHR audit logs which capture and timestamp user activity
Computer logins
Rotation schedules
Review of patient charts
Correct Answer: The correct answer is option a. As described in the Discussion, it is helpful to use objective techniques, when possible, to complement self-report measures of EHR use to accurately characterize time spent on EHR activities for future research studies. One of the most effective and speedy methods of characterizing trainee time spent on the EHR is an audit log that captures and timestamps user activity.
Acknowledgments
The authors wish to acknowledge the help of the Information and Core Services Team at Institute of Informatics at Washington University whose personnel assisted with the development of the REDCap-based survey.
Conflict of Interest T.K. reports grants from NIMH, grants from NCATS, grants from NINR, personal fees from Pfizer, outside the submitted work. B.A.E. reports grants from National Institute of Occupational Safety and Health, from null, during the conduct of the study; grants from National Institute of Occupational Safety and Health, grants from National Institute of Health, from null, outside the submitted work.
Protection of Human and Animal Subjects
This study was reviewed by the Washington University Institutional Review Board.
References
- 1.Huang J Z, Han M, Luo T, Ren A, Zhou X. Mental health survey of 230 medical staff in a tertiary infectious disease hospital for COVID-19. Chinese J Industrial Hygiene Occupat Dis. 2020;38:192–195. doi: 10.3760/cma.j.cn121094-20200219-00063. [DOI] [PubMed] [Google Scholar]
- 2.Lai J, Ma S, Wang Y. Factors associated with mental health outcomes among health care workers exposed to coronavirus disease 2019. JAMA Netw Open. 2020;3(03):e203976–e203976. doi: 10.1001/jamanetworkopen.2020.3976. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Alexander G C, Tajanlangit M, Heyward J, Mansour O, Qato D M, Stafford R S. Use and content of primary care office-based vs telemedicine care visits during the COVID-19 pandemic in the US. JAMA Netw Open. 2020;3(10):e2021476–e2021476. doi: 10.1001/jamanetworkopen.2020.21476. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Hron J D, Parsons C R, Williams L A, Harper M B, Bourgeois F C. Rapid implementation of an inpatient telehealth program during the COVID-19 pandemic. Appl Clin Inform. 2020;11(03):452–459. doi: 10.1055/s-0040-1713635. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Knighton A J, Ranade-Kharkar P, Brunisholz K D. Rapid implementation of a complex, multimodal technology response to COVID-19 at an integrated community-based health care system. Appl Clin Inform. 2020;11(05):825–838. doi: 10.1055/s-0040-1719179. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Sullivan C, Wong I, Adams E. Moving faster than the COVID-19 pandemic: the rapid, digital transformation of a public health system. Appl Clin Inform. 2021;12(02):229–236. doi: 10.1055/s-0041-1725186. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Gomez T, Anaya Y B, Shih K J, Tarn D M.A qualitative study of primary care physicians' experiences with telemedicine during COVID-19 J Am Board Fam Med 202134(Suppl):S61–S70. [DOI] [PubMed] [Google Scholar]
- 8.Barney A, Buckelew S, Mesheriakova V, Raymond-Flesch M. The COVID-19 pandemic and rapid implementation of adolescent and young adult telemedicine: challenges and opportunities for innovation. J Adolesc Health. 2020;67(02):164–171. doi: 10.1016/j.jadohealth.2020.05.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Coleman J R, Abdelsattar J M, Glocker R J. COVID-19 pandemic and the lived experience of surgical residents, fellows, and early-career surgeons in the American College of Surgeons. J Am Coll Surg. 2021;232(02):119–1.3500000000000001E22. doi: 10.1016/j.jamcollsurg.2020.09.026. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Kannampallil T G, Goss C W, Evanoff B A, Strickland J R, McAlister R P, Duncan J. Exposure to COVID-19 patients increases physician trainee stress and burnout. PLoS One. 2020;15(08):e0237301. doi: 10.1371/journal.pone.0237301. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Gallagher T H, Schleyer A M. “We Signed Up for This!”—student and trainee responses to the COVID-19 pandemic. N Engl J Med. 2020;382(25):e96. doi: 10.1056/NEJMp2005234. [DOI] [PubMed] [Google Scholar]
- 12.Li W, Frank E, Zhao Z. Mental health of young physicians in China during the novel coronavirus disease 2019 outbreak. JAMA Netw Open. 2020;3(06):e2010705–e2010705. doi: 10.1001/jamanetworkopen.2020.10705. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.De Brier N, Stroobants S, Vandekerckhove P, De Buck E. Factors affecting mental health of health care workers during coronavirus disease outbreaks: a rapid systematic review. PLoS One. 2020;15(12):e0244052. doi: 10.1371/journal.pone.0244052. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Lou S S, Goss C W, Evanoff B A, Duncan J G, Kannampallil T. Risk factors associated with physician trainee concern over missed educational opportunities during the COVID-19 pandemic. BMC Med Educ. 2021;21(01):216. doi: 10.1186/s12909-021-02665-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Arndt B G, Beasley J W, Watkinson M D. Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations. Ann Fam Med. 2017;15(05):419–426. doi: 10.1370/afm.2121. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Gupta K, Murray S G, Sarkar U, Mourad M, Adler-Milstein J. Differences in ambulatory EHR use patterns for male vs. female physicians. NEJM Catal. 2019 doi: 10.1056/CAT.19.0690. [DOI] [Google Scholar]
- 17.Gardner R L, Cooper E, Haskell J. Physician stress and burnout: the impact of health information technology. J Am Med Inform Assoc. 2019;26(02):106–114. doi: 10.1093/jamia/ocy145. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.DiAngi Y T, Stevens L A, Halpern-Felsher B, Pageler N M, Lee T C. Electronic health record (EHR) training program identifies a new tool to quantify the EHR time burden and improves providers' perceived control over their workload in the EHR. JAMIA Open. 2019;2(02):222–230. doi: 10.1093/jamiaopen/ooz003. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Shanafelt T D, Dyrbye L N, Sinsky C.Relationship between clerical burden and characteristics of the electronic environment with physician burnout and professional satisfactionPaper presented at: Mayo Clinic Proceedings2016 [DOI] [PubMed]
- 20.Tai-Seale M, Dillon E C, Yang Y. Physicians' well-being linked to in-basket messages generated by algorithms in electronic health records. Health Aff (Millwood) 2019;38(07):1073–1078. doi: 10.1377/hlthaff.2018.05509. [DOI] [PubMed] [Google Scholar]
- 21.Cox M L, Farjat A E, Risoli T J. Documenting or operating: where is time spent in general surgery residency? J Surg Educ. 2018;75(06):e97–e106. doi: 10.1016/j.jsurg.2018.10.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Bello R J, Sarmiento S, Meyer M L. Understanding surgical resident and fellow perspectives on their operative performance feedback needs: a qualitative study. J Surg Educ. 2018;75(06):1498–1503. doi: 10.1016/j.jsurg.2018.04.002. [DOI] [PubMed] [Google Scholar]
- 23.Block L, Habicht R, Wu A W. In the wake of the 2003 and 2011 duty hours regulations, how do internal medicine interns spend their time? J Gen Intern Med. 2013;28(08):1042–1047. doi: 10.1007/s11606-013-2376-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Arora A, Garg A, Arora V, Rizvi M, Desai N. National survey of pediatric care providers: assessing time and impact of coding and documentation in physician practice. Clin Pediatr (Phila) 2018;57(11):1300–1303. doi: 10.1177/0009922818774341. [DOI] [PubMed] [Google Scholar]
- 25.Mamykina L, Vawdrey D K, Hripcsak G. How do residents spend their shift time? A time and motion study with a particular focus on the use of computers. Acad Med. 2016;91(06):827–832. doi: 10.1097/ACM.0000000000001148. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Gagliardi J P, Turner D A. The electronic health record and education: rethinking optimization. J Grad Med Educ. 2016;8(03):325–327. doi: 10.4300/JGME-D-15-00275.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Wang J K, Ouyang D, Hom J, Chi J, Chen J H. Characterizing electronic health record usage patterns of inpatient medicine residents using event log data. PLoS One. 2019;14(02):e0205379. doi: 10.1371/journal.pone.0205379. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Trockel M, Bohman B, Lesure E. A brief instrument to assess both burnout and professional fulfillment in physicians: reliability and validity, including correlation with self-reported medical errors, in a sample of resident and practicing physicians. Acad Psychiatry. 2018;42(01):11–24. doi: 10.1007/s40596-017-0849-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Weiss D S. New York, NY: Springer; 2007. The Impact of Event Scale: Revised; pp. 219–238. [Google Scholar]
- 30.Henry J D, Crawford J R.The short-form version of the Depression Anxiety Stress Scales (DASS-21): construct validity and normative data in a large non-clinical sample Br J Clin Psychol 200544(Pt 2):227–239. [DOI] [PubMed] [Google Scholar]
- 31.Maslach C, Jackson S E, Leiter M P, Schaufeli W B, Schwab R L.Maslach Burnout InventoryVol. 21.Palo Alto, CA: Consulting Psychologists Press; 1986 [Google Scholar]
- 32.Brady K JS, Ni P, Carlasare L. Establishing Crosswalks Between Common Measures of Burnout in US Physicians. J Gen Intern Med. 2021 doi: 10.1007/s11606-021-06661-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Asukai N, Kato H, Kawamura N. Reliability and validity of the Japanese-language version of the Impact of Event Scale-Revised (IES-R-J): four studies of different traumatic events. J Nerv Ment Dis. 2002;190(03):175–182. doi: 10.1097/00005053-200203000-00006. [DOI] [PubMed] [Google Scholar]
- 34.Clara I P, Cox B J, Enns M W. Confirmatory factor analysis of the Depression–Anxiety–Stress Scales in depressed and anxious patients. J Psychopathol Behav Assess. 2001;23(01):61–67. [Google Scholar]
- 35.Crawford J R, Henry J D.The Depression Anxiety Stress Scales (DASS): normative data and latent structure in a large non-clinical sample Br J Clin Psychol 200342(Pt 2):111–131. [DOI] [PubMed] [Google Scholar]
- 36.Lam T P, Wong J G, Ip M S, Lam K F, Pang S L. Psychological well-being of interns in Hong Kong: what causes them stress and what helps them. Med Teach. 2010;32(03):e120–e126. doi: 10.3109/01421590903449894. [DOI] [PubMed] [Google Scholar]
- 37.Lovibond S H, Lovibond P F. Psychology Foundation of Australia; 1996. Manual for the Depression Anxiety Stress Scales. [Google Scholar]
- 38.Creamer M, Bell R, Failla S. Psychometric properties of the Impact of Event Scale—Revised. Behav Res Ther. 2003;41(12):1489–1496. doi: 10.1016/j.brat.2003.07.010. [DOI] [PubMed] [Google Scholar]
- 39.Rash C J, Coffey S F, Baschnagel J S, Drobes D J, Saladin M E. Psychometric properties of the IES-R in traumatized substance dependent individuals with and without PTSD. Addict Behav. 2008;33(08):1039–1047. doi: 10.1016/j.addbeh.2008.04.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.StataCorp L. . College Station, TX: 2019. Stata Statistical Software: Release 16.
- 41.Salazar de Pablo G, Vaquerizo-Serrano J, Catalan A. Impact of coronavirus syndromes on physical and mental health of health care workers: systematic review and meta-analysis. J Affect Disord. 2020;275:48–57. doi: 10.1016/j.jad.2020.06.022. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Axisa C, Nash L, Kelly P, Willcock S. Burnout and distress in Australian physician trainees: Evaluation of a wellbeing workshop. Australas Psychiatry. 2019;27(03):255–261. doi: 10.1177/1039856219833793. [DOI] [PubMed] [Google Scholar]
- 43.Marchalik D, Brems J, Rodriguez A. The impact of institutional factors on physician burnout: a national study of urology trainees. Urology. 2019;131:27–35. doi: 10.1016/j.urology.2019.04.042. [DOI] [PubMed] [Google Scholar]
- 44.Mahan J D. Burnout in pediatric residents and physicians: a call to action. Pediatrics. 2017;139(03):e20164233. doi: 10.1542/peds.2016-4233. [DOI] [PubMed] [Google Scholar]
- 45.Levin K H, Shanafelt T D, Keran C M. Burnout, career satisfaction, and well-being among US neurology residents and fellows in 2016. Neurology. 2017;89(05):492–501. doi: 10.1212/WNL.0000000000004135. [DOI] [PubMed] [Google Scholar]
- 46.Card A J. Physician burnout: resilience training is only part of the solution. Ann Fam Med. 2018;16(03):267–270. doi: 10.1370/afm.2223. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Dyrbye L N, West C P, Satele D. Burnout among U.S. medical students, residents, and early career physicians relative to the general U.S. population. Acad Med. 2014;89(03):443–451. doi: 10.1097/ACM.0000000000000134. [DOI] [PubMed] [Google Scholar]
- 48.Zhou A Y, Panagioti M, Esmail A, Agius R, Van Tongeren M, Bower P. Factors associated with burnout and stress in trainee physicians: a systematic review and meta-analysis. JAMA Netw Open. 2020;3(08):e2013761–e2013761. doi: 10.1001/jamanetworkopen.2020.13761. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49.Dzau V J, Kirch D, Nasca T. Preventing a parallel pandemic—a national strategy to protect clinicians' well-being. N Engl J Med. 2020;383(06):513–515. doi: 10.1056/NEJMp2011027. [DOI] [PubMed] [Google Scholar]
- 50.Galea S, Merchant R M, Lurie N. The mental health consequences of COVID-19 and physical distancing: the need for prevention and early intervention. JAMA Intern Med. 2020;180(06):817–818. doi: 10.1001/jamainternmed.2020.1562. [DOI] [PubMed] [Google Scholar]
- 51.National Academies of Sciences E ; Medicine . Washington, D.C.: National Academies Press; 2019. Taking Action against Clinician Burnout: A Systems Approach to Professional Well-Being. [PubMed] [Google Scholar]
- 52.Perry K, Gold S, Shearer E M. Identifying and addressing mental health providers' perceived barriers to clinical video telehealth utilization. J Clin Psychol. 2020;76(06):1125–1134. doi: 10.1002/jclp.22770. [DOI] [PubMed] [Google Scholar]
- 53.Kane C K, Gillis K. The use of telemedicine by physicians: still the exception rather than the rule. Health Aff (Millwood) 2018;37(12):1923–1930. doi: 10.1377/hlthaff.2018.05077. [DOI] [PubMed] [Google Scholar]
- 54.Maroongroge S, Smith B, Bloom E S. Telemedicine for radiation oncology in a post-COVID world. Int J Radiat Oncol Biol Phys. 2020;108(02):407–410. doi: 10.1016/j.ijrobp.2020.06.040. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55.CoverMyMeds 2021Medication Access ReportPublished 2021. Accessed March 1, 2021 at:https://assets.ctfassets.net/70w6ftfzv4je/5slQIbN5JquiQpuBvRA7yA/f8b16f6a0ff1d57591f50cc63d007923/CMM_77721_MedicationAccessReport21_FINAL__1_.pdf
- 56.Kannampallil T, Ma J. Digital translucence: adapting telemedicine delivery post-COVID-19. Telemed J E Health. 2020;26(09):1120–1122. doi: 10.1089/tmj.2020.0158. [DOI] [PubMed] [Google Scholar]
- 57.Romanick-Schmiedl S, Raghu G. Telemedicine—maintaining quality during times of transition. Nat Rev Dis Primers. 2020;6(01):1–2. doi: 10.1038/s41572-020-0185-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 58.Albert N M, Prasun M A. Telemedicine in heart failure during COVID-19: like it, love it or lose it? Heart Lung. 2020;49(06):A11–A12. doi: 10.1016/j.hrtlng.2020.10.014. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 59.Kahn J M. Virtual visits--confronting the challenges of telemedicine. N Engl J Med. 2015;372(18):1684–1685. doi: 10.1056/NEJMp1500533. [DOI] [PubMed] [Google Scholar]
- 60.Uscher-Pines L, Sousa J, Raja P, Mehrotra A, Barnett M L, Huskamp H A. Suddenly becoming a “virtual doctor”: Experiences of psychiatrists transitioning to telemedicine during the COVID-19 pandemic. Psychiatr Serv. 2020;71(11):1143–1150. doi: 10.1176/appi.ps.202000250. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 61.Dorsey E, Okun M S, Bloem B R. Care, convenience, comfort, confidentiality, and contagion: the 5 C's that will shape the future of telemedicine. J Parkinsons Dis. 2020;10(03):893–897. doi: 10.3233/JPD-202109. [DOI] [PubMed] [Google Scholar]
- 62.Boxer R J, Ellimoottil C. Advantages and utilization of telemedicine. mHealth. 2019;5:12. doi: 10.21037/mhealth.2019.04.02. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 63.Judson T J, Odisho A Y, Neinstein A B. Rapid design and implementation of an integrated patient self-triage and self-scheduling tool for COVID-19. J Am Med Inform Assoc. 2020;27(06):860–866. doi: 10.1093/jamia/ocaa051. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 64.Hilliard R W, Haskell J, Gardner R L. Are specific elements of electronic health record use associated with clinician burnout more than others? J Am Med Inform Assoc. 2020;27(09):1401–1410. doi: 10.1093/jamia/ocaa092. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 65.Williams M S. Misdiagnosis: burnout, moral injury, and implications for the electronic health record. J Am Med Inform Assoc. 2021;28(05):1047–1050. doi: 10.1093/jamia/ocaa244. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 66.Carayon P, Cassel C, Dzau V J. Improving the system to support clinician well-being and provide better patient care. JAMA. 2019;322(22):2165–2166. doi: 10.1001/jama.2019.17406. [DOI] [PubMed] [Google Scholar]
- 67.Lourie E M, Utidjian L H, Ricci M F, Webster L, Young C, Grenfell S M. Reducing electronic health record-related burnout in providers through a personalized efficiency improvement program. J Am Med Inform Assoc. 2021;28(05):931–937. doi: 10.1093/jamia/ocaa248. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 68.Toll E T. The other office. J Am Med Inform Assoc. 2019;26(01):71–75. doi: 10.1093/jamia/ocy144. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 69.Aarts J, Callen J, Coiera E, Westbrook J. Information technology in health care: socio-technical approaches. Int J Med Inform. 2010;79(06):389–390. doi: 10.1016/j.ijmedinf.2010.03.006. [DOI] [PubMed] [Google Scholar]
- 70.Amroze A, Field T S, Fouayzi H. Use of electronic health record access and audit logs to identify physician actions following noninterruptive alert opening: descriptive study. JMIR Med Inform. 2019;7(01):e12650. doi: 10.2196/12650. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 71.Rule A, Chiang M F, Hribar M R. Using electronic health record audit logs to study clinical activity: a systematic review of aims, measures, and methods. J Am Med Inform Assoc. 2020;27(03):480–490. doi: 10.1093/jamia/ocz196. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 72.Adler-Milstein J, Adelman J S, Tai-Seale M, Patel V L, Dymek C. EHR audit logs: a new goldmine for health services research? J Biomed Inform. 2020;101:103343. doi: 10.1016/j.jbi.2019.103343. [DOI] [PubMed] [Google Scholar]
- 73.Kannampallil T, Abraham J, Lou S, Payne P RO. Conceptual considerations for using EHR-based activity logs to measure clinician burnout and its effects. J Am Med Inform Assoc. 2021;28(05):1032–1037. doi: 10.1093/jamia/ocaa305. [DOI] [PMC free article] [PubMed] [Google Scholar]