A Comprehensive Approach for Assessing the Impact of Ambient Documentation

Abstract

Workforce challenges are a significant issue facing many healthcare organizations.¹ One variable contributing to this market dynamic is provider burnout, which remains high and is largely driven by administrative demands that continue to increase.² Healthcare organizations are rapidly adopting enabling digital capabilities, such as generative artificial intelligence (AI) technologies, that have the potential to decrease administrative burden. One such tool is ambient documentation, which aims to make clinical documentation workflows smarter and more efficient. In September 2024, ambient documentation became the first broad clinical use of generative AI at Geisinger, when the technology was deployed to 100 ambulatory providers. This paper outlines Geisinger’s evaluation and implementation approach to ambient documentation and the impact this technology has made on administrative burden, provider burnout, and patient experience.

Introduction

Burnout among healthcare providers may be at an all-time high, and organizations are rapidly deploying ambient documentation technology in response to increased administrative demands and productivity pressure.^3-6 The goal of ambient documentation is to make the note-writing workflow smarter and more efficient by harnessing automated audio transcription and generative artificial intelligence (AI) to produce a first draft of a provider’s note. In September 2024, Geisinger deployed ambient documentation (Abridge, Pittsburgh, PA) integrated within our electronic health record (Epic Corp., Verona WI) to an initial group of 100 ambulatory providers. Geisinger selected the technology option that was integrated within the EHR and optimized to support providers’ workflow.

Using quantitative and qualitative methods, we evaluated the impact of this technology on the experience of providers and patients. The learnings from the initial group has informed the ongoing rollout of the technology to our entire eligible population of primary care providers and other providers within select specialties.

Methods

The primary aim of deploying ambient documentation was to improve provider satisfaction and reduce symptoms of burn-out, which we hoped to achieve by decreasing after-hours time spent within the EHR and reducing the time providers spent writing notes for each patient.

Evaluation Approach To evaluate the impact of ambient documentation on providers and patients, we developed a comprehensive tracking dashboard in Tableau (Version 2022.3) that includes the following metrics.

Quantitative Metrics from the Electronic Health Record (EHR)

• Use of the ambient documentation technology (encounters / eligible encounters)

• Percentage of encounters (charts) closed on the day of visit

• Average minutes providers spent in EHR note-writing module per encounter

• Volume of patients seen

• Average level of service for billing

• Amount of time providers spent using the EHR outside of work hours (i.e., “pajama time”)

Qualitative Metrics Regarding Provider Experience (collected via electronic survey)

• Provider’s perception of their level of professional burnout (Mini-Z question 2)

• How likely the provider is to recommend the ambient documentation technology (using Net Promoter Score⁷)

• Provider’s perceived level of “documentation burden” measured using the NASA Task-Load Index (NASA-TLX)

• Provider’s perceived amount of time spent writing notes outside of clinic hours

• Provider’s level of confidence that patients can understand their care plans by reading the provider’s notes

• Provider’s perception that they can give patients their undivided attention during the clinical encounter

• Provider’s opinion that if urgently needed, they could add at least one additional patient encounter to their clinic session

Qualitative Metrics Regarding Patient Experience (collected via U.S. mail/email surveys)

• Perceived degree of concern the care provider showed for patient or parent/caregiver’s questions and worries

• Perception of how well the nurse/assistant listened to the patient or parent/caregiver

• Likelihood of patient or parent/caregiver recommending our practice to others

• Likelihood of patient or parent/caregiver recommending our care provider to others

• Perception of patient or parent/caregiver regarding our concern for their privacy

These metrics were determined by a project advisory committee that included key Geisinger leadership in the following areas of the organization: digital transformation, physician informatics, front line clinical leaders, revenue cycle, and finance. The committee reviewed existing literature and established measures of success for ambient documentation in accordance with the organization’s goals and priorities.

As a quality improvement endeavor, we assessed the quantitative and qualitative metrics before and after the deployment of the ambient documentation tool. For the quantitative EHR data and the qualitative patient experience data, the “pre” period included the 12 months prior to the provider beginning to use the technology, and the “post” period included the subsequent months up to the preparation of this manuscript (approaching 5 months for initial users). The patient experience surveys were administered by U.S. Mail and email using a patient experience survey vendor (Press Ganey, South Bend IN), following our organization’s existing process.

For the qualitative provider data, two surveys were administered by the team’s product manager as each round of software licenses was activated. A pre-launch survey was distributed to new users via email prior to license activation. The purpose was to establish baseline metrics for self-reported data in areas such as feelings of burnout and work outside of work. This survey included the NASA Task Load Index (NASA TLX)⁸, which has been used to measure burden in other healthcare domains, including intensive care nursing,⁹ surgical procedures,¹⁰ and recently by Balloch and colleagues¹¹ to assess ambient documentation. A post-launch survey was sent via email 30 days after each provider’s go-live to determine the impact of ambient documentation technology. Both surveys were developed in collaboration with Abridge, Inc. Surveys were created in Typeform by Abridge with raw data sent to the product team for ingestion into the metrics dashboard. Email addresses of respondents were collected to track survey completion and any responses that required follow-up. Reminders were sent via both email and secure text message. The survey also included a section for free-text comments where providers could enter feedback about the ambient documentation tool.

Implementation Approach

The pilot group of ambient documentation users was selected by the project advisory committee based on several factors. Prior to rolling out to the first 100 users, a group of ten beta users received the tool to perform user acceptance testing to assure that the product was ready for larger deployment.

To identify the first 100 users to receive licenses for the ambient documentation software, we used an “Opportunity Score” provided by the EHR vendor’s “Signal Report.” The Opportunity Score is a combination of the average number of characters typed per note, the amount of work outside of scheduled hours (pajama time) performed, and the average time to close encounter notes. It was hypothesized that providers with the highest Opportunity Score would potentially benefit the most from ambient documentation technology. Project leaders worked closely with primary care leadership to identify those providers with the highest Opportunity Scores and offer them licenses for the ambient documentation tool. This approach did not generate as many volunteers as expected, but in parallel, we received inquiries about participation from other providers who were not on the initial Opportunity Score list. Within a few weeks, we expanded our inclusion criteria to include those providers within our primary care practices who were interested in using ambient documentation regardless of their Opportunity Score.

The majority of licenses (65%) are assigned to Primary Care providers, with the remaining 35% assigned to Specialty departments. From a specialty perspective, areas of focus were orthopedics and cardiology. Most specialty areas were assigned two to four licenses to assess their use cases, with rapid expansion to specialties that were using the technology extensively. The decision to use the ambient documentation option that was integrated within our EHR created a dependency on Apple iOS. Providers who used the technology were required to use their personal iPhone or iPad device.

Beyond the initial implementation team, which included several clinical informaticists, all participants had at least a .50 FTE of clinical time. Providers were not incentivized financially or administratively to use the ambient documentation tool. Likewise, there were no requirements for them to participate (e.g., they were not asked to see more patients, pay for the tool, etc.). Software licenses were assigned on a voluntary basis, and users were free to stop using it at any time. When non-use was identified, the license was repurposed to another provider on our waiting list.

As Geisinger scales ambient documentation, operational responsibility will increase. Increasing operational input, buy-in and overall accountability will not only ensure the technology is adding value to respective clinicians with specific workflows, the responsibility of managing license usage will aid in good financial stewardship of the software.

Results

The rollout of ambient documentation technology at Geisinger has yielded substantial reduction in documentation burden, mitigating provider burnout, and enhancing the patient experience. Figure 1 shows a partial view of the comprehensive dashboard developed to assess the impact of ambient documentation. At the time of this writing, the 100 initial pilot users have been using the tool for over four months, totaling more than 40,000 encounters where ambient documentation was used (Figure 1). Eighty-one of 100 users are considered at least “moderate users” of the tool, defined as using the tool for more than 25% of their eligible visits. Forty-three users are considered “super users” based on their use of the tool for at least 75% of eligible visits. Overall, ambient documentation has been used in about two-thirds of encounters, and across all user groups (pilot and expansion) there have been over 75,000 notes generated using ambient documentation.

Figure 1.

Partial view of dashboard showing ambient documentation key performance indicators.

Compared to the one-year period prior to deploying ambient documentation, the time required for providers to generate clinic visit notes declined by an average of two minutes per visit representing an 18% reduction (Figure 2). If this figure is multiplied by number of patients seen in a provider’s typical workday (16.3 patients in the “pre” period for our pilot group), ambient documentation produces a time savings of up to 33 minutes per day. This time savings observed in the quantitative data aligns which the self-reported “work outside of work measure” collected via the provider survey, in which the average time providers spent writing notes declined from 4.1 hours to 1.9 hours per week.

Figure 2.

Monthly trend of time spent writing notes per encounter for ambient documenation pilot users.

The self-reported survey administered to the providers at baseline and at 30 days post-implementation was completed by 60 of 100 individuals (60% response rate). Respondents reported a 41% reduction in the number of providers at risk of burnout and a 63% reduction in documentation burden (Figure 3). Via this survey we observed a net promoter score for the ambient documentation tool of 71.7, representing “exceptional reputation.”

Figure 3.

Summary of provider survey results pre- and post-implementation of ambient documentation tool.

In addition to the provider survey, anecdotal feedback from providers has been very positive. Representative quotes collected from the initial pilot users include the following:

• “I have used it every chance I get–it is a gamechanger.” –Cardiology physician

• “Using this tool is amazing. It has saved me so much time.” –Primary care physician

• “I am having a wonderful experience with this technology and highly recommend continuing to roll-out.” –Orthopedic surgeon

From a patient experience perspective, the two main questions we evaluated were “Provider Concern for Questions” and “Likelihood of Recommending Care Provider.” Prior to the deployment of ambient documentation, patients seen by the pilot providers responded that provider concern was very poor or poor 1.7% of the time, with this rate dropping to 0.9% (a 47% improvement) post-implementation (Figure 5). Similarly, patients responded that they were very unlikely or unlikely to recommend their provider 2.5% of the time prior to the ambient documentation deployment; this rate dropped to 1.6% (a 36% improvement) post-implementation.

Figure 5.

Results of Patient Experience questions pre- and post-implementation of ambient documentation tool.

Discussion

Geisinger’s evaluation of its ambient documentation technology deployment yielded several noteworthy findings. First, we observed that providers’ self-reported reduction in after-hours work time significantly outpaced objective measurements in the same domain. Based on EHR audit logs, providers objectively saved about 5.3 minutes per day in after-hours work (an 18% reduction pre/post). But according to the subjective perception of providers responding to the survey, they reported a much more dramatic 2.5-hour decrease per week. This suggests that the psychological relief from documentation pressure may be even more valuable than actual time saved. This also suggests that current EHR log data is imperfect in measuring after-hours work due to a variety of assumptions (e.g., lack of keystroke activity does not always represent lack of thought related to the work activity) and confounders (e.g., inpatient clinical duties not reflected in the schedule can be misattributed to after-hours work).

Second, while the objective after-hours work decreased by only 5.3 minutes daily, the objective time spent writing notes decreased by greater than 30 minutes per day. Some of this discrepancy may be accounted for by the imperfections in the after-hours work measurement as noted above, but this may also be explained by providers redirecting their time to other electronic health record (EHR) or patient care activities.

Third, the benefit threshold for ambient documentation—i.e., frequency of use required to achieve significant benefits in provider satisfaction—was lower than anticipated, with even infrequent users (those employing the tool in less than 25% of encounters) reporting meaningful improvements. For example, we observed that most of our pediatricians used the tool only for about 25% of their visits, but they still reported considerable improvements in outcomes. They attributed this to the fact that 75% of their encounters are simple templated well-child visits, but they used the tool on the 25% of their visits which represent acute care and contribute a disproportionate share to their cognitive and administrative burden. This contradicts the assumption that ambient documentation requires high adoption to deliver value and indicates that strategic deployment in selected specialties and encounter types may still yield significant benefits.

Lastly, our predefined benefit categories often failed to predict which users would experience the greatest improvements. Some specialists with extensive unstructured documentation requirements reported minimal benefit, while some providers with highly templated workflows who used the tool sparingly still experienced substantial improvements. Anecdotally, some users reported that they did not believe they had much opportunity to improve documentation efficiency prior to using the tool; however, even despite that hesitancy, many ended up seeing similar efficiency gains to those who were initially more excited to adopt the tool. These findings underscore the importance of evaluating ambient documentation tools through multiple lenses rather than relying on singular metrics or predetermined user categories.

Throughout the pilot, we recognized that the benefit gained from ambient documentation technology can vary across users’ specialty and personal workflows and technology aptitude. In fact, we noted several common user types whose particular experience with ambient technology might be misunderstood if viewed through the wrong lens. Highly efficient note-writers who complete documentation promptly but sacrifice patient engagement elements like eye contact and natural conversation may experience minimal time savings but substantial improvements in both physician and patient experience. Conversely, providers who prioritize patient engagement and consequently fall behind on documentation could see significant reductions in after-hours work without necessarily perceiving better patient interactions.

Additionally, while detail-oriented providers who value their specific documentation style may be unsatisfied by the draft produced by ambient documentation, even those note-rewriters who thoroughly revise their notes may still appreciate having an initial draft produced automatically, which allows them transition between patients without fear of forgetting critical information.

There are many elements we might miss in an original concept of who might most benefit from ambient documentation technology, which made it more valuable that we deferred to choosing pilot users based on their degree of interest rather than our external evaluation of whether they would benefit. This, combined with a readiness to adjust access and to avoid making those adjustments punitively, allowed us to quickly mobilize this tool to users for whom it would be most effective.

Limitations and Considerations

Several important barriers and risks warrant consideration. Our implementation was constrained by this technology’s initial availability exclusively on iOS devices, creating an access barrier that some providers overcame by purchasing personal devices. While future updates may resolve this limitation, the hardware dependency represents an ongoing challenge for equitable implementation.

A more fundamental concern involves the educational implications of this technology.¹² Many medical educators consider note-writing an essential skill for independent practice, not merely a clerical task, but a crucial element of medical reasoning. The process of creating clinical documentation provides a structured opportunity for providers to synthesize patient information and develop diagnostic reasoning. Consequently, deploying this technology to medical learners raises questions beyond time efficiency. While Geisinger has not yet extended ambient documentation to learners, the rapid adoption of this technology across healthcare suggests a need to prepare trainees for its use, including planning for alternative methods to support their clinical reasoning.

Related to the education concerns noted above, there is a concern that certain aspects of traditional note-writing promote synthesis and drive understanding and decision making by even experienced providers. In other words, it is likely that some elements of clinical note-writing add value to care delivery.¹² The extent to which this is a true problem and if so whether it can be solved for by thoughtful review and editing or using ambient tools only for certain portions of the note such as history and exam but not assessment remains to be seen.

Financial cost is another significant consideration for deploying ambient documentation. The substantial cost of this technology has prompted many organizations to seek productivity-related offsets to justify implementation. Our approach deliberately prioritized provider satisfaction through decreased after-hours documentation and reduced burden rather than establishing productivity quid pro quos. Though we anticipate potential downstream cost savings through reduced turnover and potentially increased appointment capacity, these were secondary considerations. Importantly, many providers expressed concern about potential connections between this technology and productivity expectations, suggesting that organizational focus on productivity gains may undermine the substantial wellbeing benefits. For this reason and given the dramatic improvements in provider experience we observed, Geisinger will continue to prioritize provider wellbeing as our primary metric of success rather than productivity outcomes.

Conclusion

The quantitative and qualitative assessment of our ambient documentation deployment has been helpful to understand the impact of the technology on providers and patients, and to help inform the next steps in deployment. Our results demonstrated substantial benefits to providers’ note-writing efficiency and reduced levels of frustration and burnout. We also noted many providers indicating a wariness regarding how this technology may be connected to or made dependent on productivity.

Figures & Tables

Figure 4.

Breakdown of three questions asked to assess documentation burden pre- and post-implementation of ambient documentation tool.