Ambient artificial intelligence scribes: physician burnout and perspectives on usability and documentation burden

Abstract

Objective

This study evaluates the pilot implementation of ambient AI scribe technology to assess physician perspectives on usability and the impact on physician burden and burnout.

Materials and Methods

This prospective quality improvement study was conducted at Stanford Health Care with 48 physicians over a 3-month period. Outcome measures included burden, burnout, usability, and perceived time savings.

Results

Paired survey analysis (n = 38) revealed large statistically significant reductions in task load (−24.42, p <.001) and burnout (−1.94, p <.001), and moderate statistically significant improvements in usability scores (+10.9, p <.001). Post-survey responses (n = 46) indicated favorable utility with improved perceptions of efficiency, documentation quality, and ease of use.

Discussion

In one of the first pilot implementations of ambient AI scribe technology, improvements in physician task load, burnout, and usability were demonstrated.

Conclusion

Ambient AI scribes like DAX Copilot may enhance clinical workflows. Further research is needed to optimize widespread implementation and evaluate long-term impacts.

Background and significance

Electronic health record (EHR) documentation burden significantly contributes to physician burnout.^1–11 Strategies to reduce this burden include the 2021 evaluation and management coding changes,¹² medical scribes to assist clinicians with EHR documentation,^13,14 and automatic speech recognition technologies.^15,16 The integration of artificial intelligence (AI), particularly generative AI,¹⁷ presents an opportunity to further mitigate clinical documentation burden. Ambient AI scribes powered by generative AI using large language models (LLMs) can be leveraged to generate clinical notes within minutes after a clinic visit.¹⁸ A 2023 survey of healthcare executives identified ambient AI scribes as one of the top use cases for generative AI in healthcare.¹⁹ Early findings suggest ambient AI scribes may improve clinician efficiency and reduce clinician burnout.¹⁸ Given the high costs of these technologies, further real-world evaluations are needed²⁰ to understand the potential benefits and challenges, inform iterative improvements for technology developers, and guide organizations on scaling these solutions broadly.

Objective

To evaluate the pilot implementation of an ambient AI scribe technology to assess physician perspectives related to usability, perceived utility, and the impact on physician burden and burnout.

Methods

Study design and setting

This prospective quality improvement study was conducted from October 2023 to January 2024 at Stanford Health Care, an academic medical center in Northern California, encompassing both faculty and community practice networks.

Technology implementation and physician training

Stanford Health Care collaborated with Epic Systems Corporation (Epic) and Nuance Communications to integrate DAX Copilot ambient AI scribe technology into clinical workflows. The tool was integrated within the EHR, enabling clinicians to record conversations directly from the EHR mobile application, Epic Haiku. Draft note content was parsed into 4 auto-populated, customizable sections called Epic SmartSections: history of present illness, physical exam, results, and assessment and plan. These SmartSections could be independently embedded into existing EHR note templates, allowing flexibility to use these with other note creation tools such as free text, predefined blocks of text (Epic SmartPhrases), and speech recognition tools.²¹ Once SmartSections were embedded into the EHR note template, AI generated draft content would automatically populate within a few minutes of concluding the recording. Additionally, an attestation SmartSection was embedded into the note template to documented that patients had been informed of the recording and that the note had been reviewed by the clinician.

Multimodal training resources were developed including knowledge base articles, meeting presentations, and training sessions. All physicians were required to participate in training. Individual and group training sessions for onboarding support were conducted early in the pilot. Additional ad hoc support was offered through weekly support sessions and 1:1 assistance available upon request. Training materials included instructions on how to record with the mobile application and integrate SmartSections into preferred note templates. Physicians were instructed to notify patients about being recorded prior to initiating a visit.

Pilot recruitment

Physicians were recruited from both faculty and community practice settings across primary care and ambulatory specialties through a combination of purposive and convenience sampling. Exclusion criteria for the pilot included concurrent documentation support with a medical scribe and physicians without access to an Apple smartphone. Physicians were recruited in a phased approach from September to November 2023 until the target (n = 50) was met. Physicians who expressed interest and/or had higher direct patient care time and above average time spent in notes were targeted for recruitment. Potential physician participants were invited to participate in the pilot on a voluntary basis via email. Licenses for individual physicians to use the DAX Copilot tool were reallocated for 2 physicians who voluntarily opted out or stopped using the tool after the first 4 weeks of onboarding. A total of 48 physicians remained in the pilot and contributed data to the present analysis. Licenses were initially allocated for 3 months, with the option to extend after the end of the pilot period.

Data collection and study measures

Pre-surveys and post-surveys were emailed to all participating physicians prior to their DAX Copilot activation (pre-survey) and 3 months after activation (post-survey). Multiple reminder emails were sent to increase the yield of survey response for both pre and post-surveys. Physicians who completed the presurvey or postsurvey were compensated with a $20 gift card upon completion of the postsurvey.

The RE-AIM/PRISM framework was used to evaluate the implementation from the perspective of physicians.^22–25 Pre-survey demographics included practice type, age range, gender, years post completion of residency/fellowship training, and number of half days spent in clinic per week. Outcome measures included burden, burnout, and perceived time savings.

EHR burden was evaluated using the 4-item physician task load (PTL) index derived from an adapted NASA TLX. Task load scores range from 0 to 100, with lower scores indicating less cognitive task load. Burnout was evaluated using the 4-item work exhaustion (PFI-WE) sub-scale of the Stanford Professional Fulfillment Index (PFI)²⁶ using the standard, published approach followed by normalization of the scale to a 0- to 10-point scale for ease of interpretation,²⁷ with lower scores indicating lower levels of WE. Burden and burnout scores were collected in both pre and post surveys.

Perceived time savings was measured in the post-survey with a scale ranging from 90 minutes saved to 90 minutes of additional time spent per half day of clinic. Physician perspectives on the intervention focused on usability and physician acceptability. Usability was assessed using the 10-item System Usability Scale (SUS).^28,29 The pre-survey asked physicians to reflect on current clinical documentation resources while the post-survey asked for reflections on DAX Copilot. Physicians’ ratings of the acceptability of DAX Copilot was examined in the post-survey using an adapted and shortened version of the Unified Theory of Acceptance and Use of Technology (UTAUT) questionnaire. Sustainability and maintenance infrastructure was measured with a post-survey question about use of DAX Copilot in long-term practice.

Statistical analysis

Pre versus post-survey analysis

The analysis of survey responses used either the paired t-test or the Wilcoxon signed-rank test, and statistical significance was considered for 2-sided P < .05. Cohen’s d effect size was calculated for mean differences between PLX, PFI, and SUS. Physicians with incomplete or missing presurvey or postsurvey data were excluded from the paired analysis. Proportions and counts were used to describe all postsurvey results. Survey data were analyzed using Minitab Statistical Software version 21.4.2.0 (Minitab, LLC).

Post-survey analysis

Descriptive analysis was used for post-survey questions pertaining to physician acceptability, sustainability, and maintenance infrastructure. Each item on the survey was scored individually and the proportion of responses for each specific category (eg, “strongly agree”) was calculated by dividing the number of responses in that category by the total number of respondents for that survey item. “Strongly agree” and “agree” responses were combined to report survey results.

Ethical approval

The Stanford University institutional review board (IRB) office determined that this study met the criteria for quality improvement and was exempt from IRB-mandated consent. Standards for Quality Improvement Reporting Excellence (SQUIRE) reporting guidelines for quality improvement studies were followed.

Results

Of the 48 physicians who remained enrolled in the pilot (median use of 12.8 weeks), 38 were included in the paired pre and post-survey analysis and 46 in the unpaired post-survey analysis.

Pre versus post survey results

The majority of physicians included in the paired pre and post-survey analysis (n = 38) were from community practices (61%) rather than faculty practices (39%), and primary care (66%) rather than ambulatory specialties (34%) (Table 1). Reported mean number of half days of clinic per week was 6.3 (range 0-10).

Table 1.

Physician characteristics (n = 38).

Characteristic	Group	Count N (%)
Community practice		23
	Primary care	18
	Ambulatory specialtya	5
Faculty practice		15
	Primary careb	7
	Ambulatory specialtyc	8
Age range	25-34	7
	35-44	7
	45-54	15
	55-64	7
	≥65	2
Gender	Female	22
	Male	15
	Non-binary	0
	Prefer not to answer	1
Years after training	0-4	10
	5-9	3
	10-14	10
	≥15	15
Reported mean half days of clinic per week
	<2.5	2
	Between 2.5 and <5	7
	Between 5 and <7.5	13
	>7.5	16

^aIncludes cardiology, gastroenterology, and rheumatology.

^bIncludes geriatrics sub-specialty.

^cIncludes cardiology, otolaryngology, neurology, ophthalmology, and rheumatology.

We observed a large statistically significant reduction (−24.42 on 0-100 scale; p <.001) in the 4-item physician task-load score derivative and burnout (−1.94 on 0-10 scale; p <.001) assessed via the 4-item work exhaustion score (PFI-WE; Table 2). When comparing the usability of baseline current-state clinical documentation resources to the usability of DAX Copilot, a moderate statistically significant improvement (+10.92 on 0-100 scale, p <.05) in the SUS score was found.

Table 2.

Pre versus post survey comparison results (n = 38).

Physician task load (PTL), Burnout, system usability score (SUS)
Metric	Pre-pilot mean (SD)	Post-pilot mean (SD)	Cohen’s d	T statistic	P-value
PTL	68.98 (17.35)	44.56 (20.62)	1.28	7.35	.000
Burnouta	7.47 (2.41)	5.53 (2.10)	0.86	5.12	.000
SUS	58.09 (15.37)	69.01 (16.24)	0.69	−3.20	.003

Utility and utilization
Questionb	Pre-pilot mean (SD)	Post-pilot mean (SD)	Cohen’s d	Wilcoxon statistic	P-value
“I [plan to] use DAX Copilot often when completing my documentation”	1.61 (0.72)	2.16 (1.41)	0.49	166	.005
“I [will] have the necessary support to use DAX Copilot effectively.”	1.89 (0.80)	1.47 (0.56)	0.61	10	.008
“DAX copilot [will be/is] easy to learn how to use”	2.08 (0.85)	1.47(0.56)	0.85	17	.001
“DAX Copilot [will improve/improves] my efficiency with documentation tasks.”	1.79 (0.74)	2.21 (1.23)	0.41	185	.060
“DAX Copilot [will improve/improves] the quality of my clinical documentation.”	2.21 (1.07)	2.47 (1.16)	0.23	142	.173
“My colleagues [will be/are] interested in using DAX Copilot.”	1.79 (0.81)	1.74 (0.12)	0.09	68.50	.723
“I can see myself using DAX Copilot in my practice long-term.”	1.84 (0.82)	1.95 (1.23)	0.11	69	.629

^aAs assessed by the work exhaustion sub-scale of the Stanford PFI.

^bResponse options of strongly agree (1), agree (2), neutral (3), disagree (4), and strongly disagree (5).

Physicians expressed overall optimism about utility and utilization before the pilot, and these perceptions remained largely unchanged afterward. Physician perceptions regarding the tool’s ease-of-use increased (−0.61 on 1-5 scale) (p <.05), as did their assessment of whether they had sufficient support to use the tool effectively (−0.42 on 1-5 scale) (p <.001).

Post survey results

Among 46 post survey respondents, a majority reported favorable utility with DAX Copilot, specifically related to improved efficiency with documentation tasks (65%), improved quality of clinical documentation (52%), and ease of use (98%) (Figure 1). Approximately 65% reported using DAX Copilot often when completing their documentation, 80% reported that their colleagues were interested in using DAX Copilot, and 78% reported that they could see themselves using DAX Copilot in their practice long-term.

Figure 1.

Post Survey Likert scale results (n = 46).

The median perceived time savings per half day of clinic when using DAX Copilot was 20 minutes (range 90 minutes saved to 90 additional minutes spent) (Figure 2).

Figure 2.

Post survey results of perceived time savings (n = 46).

Discussion

In one of the first pilot implementations of ambient AI scribe technology, physician perspectives related to usability, perceived utility, and the impact on physician burden and burnout were assessed. Overall, pilot users found the usability of DAX Copilot to be better than the usability of their current-state clinical documentation resources as measured by the SUS. Notably, there was a large statistically significant reduction in burden measured by PTL score and burnout measured by WE score. Among the pilot users who completed the post survey, we found improved usability and perceived utility across domains, including improved efficiency with documentation tasks, improved quality of clinical documentation, and ease of use. Based on the positive outcomes of the pilot, licenses were extended for all participating physicians, with the majority continuing to use the tool beyond the initial 3-month period. We plan to assess long-term utilization patterns in future studies.

Limitations include the modest sample size as there were only 50 ambient AI scribe licenses available for this pilot. A rolling recruitment strategy was used, and a small number of licenses (n = 2) were re-allocated to new pilot users during the study, leading to different amounts of exposure to the ambient AI scribe tool during the evaluation interval. Physicians who expressed interest in using the tool were included, which may have introduced selection bias, favoring those more likely to report positive views of the tool. Other aspects of the study may be considered strengths. Taskload, usability, and burnout were assessed using standardized instruments. Participants in the pilot had a range of direct patient care time, and included both community and faculty physicians as well as primary care and ambulatory specialty physicians. Our post survey response rate was high (46/48; 95.8%) and 38 of 48 (79.2%) participants had paired data from both the pre and post survey timepoints.

We identified several challenges related to the implementation of ambient AI scribe technology, including recruitment, integration into existing workflows, heterogeneity of fit, and rapid technological evolution. Lessons learned from these challenges may help inform strategies for scaling ambient AI scribe technology. Understanding the heterogeneity of ambient AI scribe utilization highlights the need to identify which physicians may benefit most, especially given the high cost and resource utilization of these tools. Another lesson learned is the importance of setting clear expectations about ambient AI scribes including that irrespective of time savings, clinicians may find other benefits, such as improved efficiency, documentation quality, and physician-patient engagement. Although several physicians self-reported increased documentation time, this may have been due to an initial learning curve, early hypervigilance, or repurposing time spent writing into time spent editing. We hypothesize that both perception of and actual time spent are likely to change with ongoing use and technological advances, such as enhanced workflow integration (eg, assisting with order entry or patient instructions). A final lesson learned is that clinician feedback is essential for driving technological improvements that maximize utility for a greater number of clinicians across clinical contexts. Minor updates were made to the DAX Copilot tool during the pilot, such as addressing recording interruptions from incoming phone calls and improving pronoun capture. These enhancements, directly informed by pilot participant feedback, highlight the iterative nature of the tool’s development.

Future assessments should incorporate qualitative interviews of clinicians and mixed-methods approaches that triangulate findings with thematic interview analyses and quantitative utilization data. Future work should also qualitatively assess patient perspectives and preferences, and evaluate for any potential differences in the use of ambient AI scribes across diverse patient groups. This work can help ensure an inclusive and equitable approach to the development and implementation of ambient AI scribes in clinical care.

Conclusion

A novel pilot implementation of ambient AI scribe technology demonstrated a reduction in physician burden and burnout, along with improved usability and perceived utility compared to previously available clinical documentation resources. These findings suggest that ambient AI scribes have the potential to enhance clinical workflows and improve the experience of clinical documentation. The substantial costs of these technologies necessitate a thorough return on investment (ROI) analysis before they can be widely adopted. This pilot highlights the importance of incorporating physician perspectives on usability, perceived utility, and the impact on physician burden and burnout into the ROI analysis for ambient AI scribes. Further research is needed to assess quality, safety, and patient experience. Adaptive implementation strategies, with continuous feedback loops between clinicians and developers, will be essential to refine and enhance the utility of AI ambient scribe technology. Future research should involve larger, more diverse cohorts and employ mixed-methods approaches that incorporate patient perspectives to comprehensively evaluate the utility and impact of these tools. As the adoption of ambient AI scribes grows, responsible AI stewardship with ongoing monitoring infrastructures will also be necessary to ensure responsible and safe use.

Author contributions

Shreya J. Shah (Corresponding author), Shreya J. Shah, Stephen P. Ma, Margaret Smith, Clarissa Delahaie, Caroline Hsia, Tait Shanafelt, Michael A. Pfeffer, Christopher Sharp, Steven Lin, Patricia Garcia (Conceptualization), Shreya J. Shah, Anna Devon-Sand, Margaret Smith (Project administration), Shreya J. Shah, Anna Devon-Sand, Stephen P. Ma, Liang, Clarissa Delahaie, Caroline Hsia, Tait Shanafelt, Steven Lin, Patricia Garcia (Methodology), Clarissa Delahaie, Caroline Hsia, Michael A. Pfeffer, Christopher Sharp, Patricia Garcia (Resources and Software), Shreya J. Shah, Anna Devon-Sand, Stephen P. Ma, Yejin Jeong, Trevor Crowell, Caroline Hsia (Data curation, formal analysis, and visualization), Shreya J. Shah, Anna Devon-Sand, Patricia Garcia (Writing of the original manuscript), Shreya J. Shah, Stephen P. Ma, Yejin Jeong, Trevor Crowell, Margaret Smith, Liang, Clarissa Delahaie, Caroline Hsia, Tait Shanafelt, Michael A. Pfeffer, Christopher Sharp, Steven Lin, Patricia Garcia (Review and editing of the original manuscript), Shreya J. Shah, Margaret Smith, Tait Shanafelt, Michael A. Pfeffer, Christopher Sharp, Steven Lin, Patricia Garcia (Supervision).

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Conflicts of interest

T.S. reported having a patent for Well-being Index Instruments and Mayo Leadership Impact Index, with royalties paid from Mayo Clinic, and receiving honoraria for presenting grand rounds and keynote lectures and advising health care organizations on clinician well-being. All other authors have no competing interests to declare.

Data availability

The data underlying this article will be shared on reasonable request to the corresponding author.