Ambient artificial intelligence scribes: utilization and impact on documentation time

Abstract

Objectives

To quantify utilization and impact on documentation time of a large language model-powered ambient artificial intelligence (AI) scribe.

Materials and Methods

This prospective quality improvement study was conducted at a large academic medical center with 45 physicians from 8 ambulatory disciplines over 3 months. Utilization and documentation times were derived from electronic health record (EHR) use measures.

Results

The ambient AI scribe was utilized in 9629 of 17 428 encounters (55.25%) with significant interuser heterogeneity. Compared to baseline, median time per note reduced significantly by 0.57 minutes. Median daily documentation, afterhours, and total EHR time also decreased significantly by 6.89, 5.17, and 19.95 minutes/day, respectively.

Discussion

An early pilot of an ambient AI scribe demonstrated robust utilization and reduced time spent on documentation and in the EHR. There was notable individual-level heterogeneity.

Conclusion

Large language model-powered ambient AI scribes may reduce documentation burden. Further studies are needed to identify which users benefit most from current technology and how future iterations can support a broader audience.

Background and significance

Electronic health record (EHR) documentation burden is a significant contributor to physician burnout.^1–8 Strategies to reduce documentation burden, including team documentation,^9–13 automatic speech recognition technologies,¹⁴ and 2021 E&M coding changes¹⁵ have been implemented with variable impact.¹⁶ The development of generative AI¹⁷ has given rise to a new wave of ambient AI scribes powered by large language models (LLMs). Initial versions of this technology utilizing human scribes to review AI output and manual copy-paste workflows to integrate content into the EHR were promising.¹⁸ A subsequent study of a fully autonomous LLM-powered ambient AI scribe without EHR integration reported relative time savings; however, uptake was modest, the time analysis was restricted to primary care physicians, and absolute time savings were not reported.¹⁹ Newer fully autonomous and EHR integrated versions that can generate drafts of clinical notes within seconds have even greater potential to effect large-scale change but require further evaluation.

Objectives

To better understand the utility and quantify the time impact of an ambient AI scribe, DAX Copilot (Nuance Communications, Inc.) was deployed in select ambulatory disciplines at an academic medical center. Primary outcome measures included utilization and documentation time per note. Secondary measures included daily documentation time, afterhours EHR time (defined as time spent in the EHR outside of weekdays between 7 AM and 5:30 PM) and total EHR time.

Materials and methods

Setting

The intervention was piloted during a 3-month period from October 2023 to January 2024 in ambulatory settings at a single academic medical center (Stanford Health Care).

Technology integration

Stanford Health Care collaborated with EHR developer Epic (Epic Systems) and Nuance (Microsoft Corporation) to integrate DAX Copilot into clinical documentation workflows. The EHR’s mobile app is used to record the physician-patient interaction. The resulting transcript is then processed by an LLM, which generates drafts of 4 note sections that can be accessed via predefined shortcut phrases (SmartSections): history of present illness (HPI), physical exam (PE), results, and assessment and plan (A&P). Physicians insert one or more of these SmartSections into their existing note templates, which then autopopulate once the recording is complete (Figure S1).²⁰

Study recruitment

Physicians were recruited through convenience and purposive sampling based on interest and suspected documentation burden between August 2023 and December 2024. Exclusion criteria included existing in-person or virtual scribe support, working exclusively with trainees, or lacking access to an iPhone (Apple, Inc.). Users were onboarded in a rolling fashion up to a 50-license limit. Users who perceived that the tool had low utility were allowed to voluntarily relinquish their license for reallocation to a nonstudy participant and were excluded from further analysis. Onboarded users who did not have ambulatory patient encounters during the baseline prepilot period or the pilot period were also excluded from analysis.

Physician training

Pilot users underwent required training prior to gaining access to DAX Copilot. Training consisted of a live onboarding support session where SmartSections were added to existing note templates and mobile app workflows were reviewed. This 30-minute live training was subsequently transitioned to a shorter, asynchronous, self-guided video lesson. Multimodal support materials, including knowledge base articles, demo videos, and drop-in group sessions were also made available to pilot users.

Study analyses

All ambulatory encounters (including procedural and virtual encounters) performed by pilot physicians who had access to the tool for at least 30 days between October 29, 2023 and January 27, 2024 were included in the analysis. The initial version of DAX at the beginning of the study period was 10.6.4 and the final version at the end of the study period was 10.7. Encounters where the patient’s preferred language was not English were excluded. The analysis period dates were selected based on audit reporting periods predefined by the EHR. The tool was available to 21 pilot users prior to the analysis start date for technical troubleshooting. All remaining pilot users were enrolled after October 29, 2023, and the entirety of their pilot experience is represented in the analysis period. Where relevant, outcomes from the pilot period were compared against a 2-month baseline prepilot period from May 28, 2023 to July 29, 2023.

Study measures

All outcomes were based on EHR use measures sourced directly from Epic. All measures were summarized as both means and SD as well as medians and interquartile ranges (IQRs).

Statistical analysis

Physician demographics were described with counts and proportions.

Utilization was calculated as the fraction of encounters seen by the physician for which DAX was utilized and was similarly calculated for the use of individual SmartSections.

Electronic health record audit logs (Epic Signal) were only available as averages for each physician across a reporting period rather than at the level of individual encounters. To calculate the average note time per physician in the prepilot period, the average note time for that physician across all prepilot reporting periods was weighted by the number of notes written during the associated reporting period. Weighted averages were similarly calculated for the pilot period, and the difference in average times between the prepilot and pilot periods was calculated for each physician. A one-sample t-test was used to compare the observed differences against a null hypothesis of no change between the 2 periods. A P-value<.05 was considered significant. A similar analysis was performed for average daily documentation time, afterhours EHR time, and total EHR time, but weighting was performed by the number of days in the study period.

Analysis was performed using the statsmodels package in Python programming language version 3.11.4 (Python Software Foundation).

Ethics approval

The Stanford University Institutional Review Board determined that the project was exempt from formal institutional review board oversight on the basis of quality improvement.

Results

Cohort

Of the 50 onboarded users, 3 relinquished their license due to low perceived utility and 2 were excluded for lack of encounters in either the pilot or baseline periods; 45 were included in the study analysis (Figure S2) representing 8 specialties (Table 1). The majority of pilot users were female, 10 or more years beyond training, and specialized in primary care. The full range of ambulatory practice time, from 1 to 10 half days of clinic per week, was represented in the pilot population (Table 1), with a mean of 6.2 half days per week. Each physician saw a median of 298 encounters in the prepilot period (IQR 202 to 411) and 338 encounters in the pilot period (IQR 200 to 517) (Figure S3 and Table S1).

Table 1.

Pilot user demographics.

Characteristic	Counts (Percentages)
Overall cohort	45 (100%)
Gender category
Female	28 (62%)
Male	17 (38%)
Nonbinary	0 (0%)
Years after training
0-4	11 (24%)
5-9	3 (7%)
10-14	14 (31%)
≥15	17 (38%)
Half days of ambulatory practice per week
1-2.5	4 (9%)
>2.5-5	11 (24%)
>5-7.5	12 (27%)
>7.5-10	18 (40%)
Specialty
Primary Care	25 (56%)
Rheumatology	7 (16%)
Otolaryngology	4 (9%)
Cardiology	3 (7%)
Gastroenterology	3 (7%)
Geriatrics	1 (2%)
Ophthalmology	1 (2%)
Orthopedics	1 (2%)

Utilization

Of the 17 428 encounters in the analysis period, DAX Copilot was utilized for 9629 encounters (55.25%). At the level of individual physicians, median utilization was 52.5% (IQR 17.86% to 80.97%), with higher utilization of the HPI and A&P SmartSections (median 39.7% and 14.8%, respectively) than the Results and PE SmartSections (median 2.6% and 1.6%, respectively) (Figure 1 and Table S2). There was significant interuser heterogeneity in utilization, with some physicians utilizing DAX for almost all encounters and others having minimal utilization (Figure 1 and Table S3).

Figure 1.

Utilization of DAX Copilot and individual SmartSections across physicians.

EHR time

The majority of physicians spent less time on clinical documentation during the pilot period compared to the prepilot period as measured using Epic Signal (Figure 2), with a median change in time per note of −0.57 minutes (IQR −1.3 to −0.13) (Table S4). This change was statistically significant (T-statistic −5.19, P<.001). The median user average time per note during the prepilot period was 4.86 minutes (IQR 2.89 to 6.47), which dropped to 3.64 minutes in the pilot period (IQR 2.45 to 5.46). To enable additional comparisons below, we also analyzed daily documentation time (Figure 3A and Table S5) and identified a median decrease of −6.89 minutes (IQR −22.37 to −0.65), which was also statistically significant (T-statistic −4.48, P<.001).

Figure 2.

Change in average note time during ambient AI scribe pilot across physicians.

Figure 3.

Change in (A) average daily documentation time, (B) average daily afterhours time, and (C) average daily total EHR time during ambient AI scribe pilot across physicians. Abbreviation: EHR, electronic health record.

There were also statistically significant decreases in daily afterhours EHR time (T-statistic −2.65, P = .01) and daily total EHR time (T-statistic −5.85, P<.001) with median changes of −5.17 minutes (IQR −21.32 to 3.82) and −19.95 minutes (IQR −39.34 to −3.64), respectively (Figure 3B and C, and Tables S6 and S7).

Discussion

In one of the first pilot studies to quantify utilization and time savings across multiple specialties, ambient AI scribes were widely used and associated with reduced documentation and EHR times. These effects were seen in physicians from a variety of clinical specialties, highlighting the broad applicability of this technology. That said, there was notable individual variability in how physicians used the tool, with some preferring to return to preexisting documentation workflows. In terms of time savings, the absolute average time saved was modest, and there were a small number of users who spent more time per note compared to baseline. This heterogeneity implies that certain user phenotypes may derive more utility from current ambient AI technology than others.

Limitations of this study include small sample size, volunteer and selection bias, potential impact from secular trends, inability to analyze time outcomes at the level of individual patient encounters, lack of comparison to alternate strategies, limitations to English-speaking patients only, and predominance of primary care physicians in our cohort. Additionally, EHR use measures have known limitations when quantifying absolute changes.²¹ In particular, afterhours EHR time and total EHR time were calculated per calendar day without normalization to the number of encounters seen, so they may be underestimating the impact of this technology particularly given the prevalence of clinic half-days in our organization. The use of a 5-second inactivity timer likely also underestimates both total documentation time as well as the absolute magnitude of change in documentation time.

Future opportunities for research in this area include evaluating the quality of notes written with ambient AI scribe assistance particularly with respect to medical synthesis and clinical communication, understanding how physicians review and edit generated text, assessing physicians’ overall perceptions about the costs and benefits of AI scribes, understanding the impact on the patient-physician relationship, and optimizing tool performance and workflows to maximize benefits.

Conclusion

In this pilot study of an LLM-powered ambient AI scribe, there was robust adoption across multiple specialties with modest reductions in documentation and EHR times. These findings highlight the potential of ambient AI scribes to reduce EHR documentation burden and mitigate physician burnout. However, the technology is not yet a one-size-fits-all solution. To reach its full potential, ambient AI scribe technology will need to evolve to effectively support the diverse needs and practices across the medical landscape.

Author contributions

All authors contributed to the design of the study, analysis of the results, drafting and revising the article, and the final approval of the submitted version.

Supplementary material

Supplementary material is available at Journal of the American Medical Informatics Association online.

Funding

None declared.

Conflicts of interest

None declared.

Data availability

The data underlying this article will be shared on reasonable request to the corresponding author.