Please document the nature of the medical emergency

In 1995, Star Trek Voyager debuted with its vision of a fully autonomous, artificial intelligence (AI) powered physician (The EMH-Emergency Medical Hologram), which was thrust into the role due to loss of the supervisory human in the loop practioner. This vision of the future has nearly arrived. AI has been integrated into multiple aspects in healthcare, with its uptake rapid, and in many cases, outpacing the science behind its impact on safety and care quality. Nowhere has its impact been more dramatically seen as in documentation support. Over the last 20 years, there has been growing attention to the problems with clinician burnout and the role documentation burden plays in it. ¹ The maturity of Large Language Models (LLM), especially when combined with ambient speech recognition (ASR), has led to a myriad of tools to facilitate documentation in multiple settings. Their ease of use and high demand has led to their rapid proliferation. Even as early as 2024, nearly 90 % of surveyed institutions had or were planning on adopting AI, with highest use in Predictive models, Radiology and Ambient Documentation. ^{2 3} With this growth, it is imperative that a standard methodology be established to measure the impact of AI generated output, on not only provider-centric measures, but patient-centric ones as well, including safety, and quality.

The paper by Toker and colleagues in this issue of BMJ Quality and Safety, ⁴ describes how various AI generated outputs are is being used in radiology to help with triage of radiologic procedures (based on urgency), quantitative analysis (such as calcium scoring), automated detection and use of predictive algorithms. A fundamental issue with these systems is that they often live separate from the EHR so providers are copying and pasting the AI generated output into unstructured notes which can lack important contextual information resulting in negative downstream impacts on patient care. Further, even with full integration, it would be unclear which parts of a final radiology report were AI generated versus human generated, thus making it difficult, if not impossible, for practioners to effectively vet the accuracy of AI-generated output, or being the “human in the loop”. The authors describe how unstructured AI output automatically layered onto a non-standard human driven workflow can lead AI to serve as a force-multiplier for medical error, including diagnostic error and inappropriate utilization of radiology resources. This is driven by the lack of a standardized reporting template and the authors propose the use of a structured reporting (SR) and guided reporting (GR) template. SR provides a set structure for what AI generated content should be imported and where, preserving metadata and attribution, while GR imbeds context sensitive decision logic, unique to the study and patient. The authors posit that adoption of a SR/GR reporting structure will allow for clinicians to achieve maximal benefit from AI, while maintaining professional responsibility, ensuring transparency and improve overall care quality.

This article highlights the promise and pitfalls to the rapid proliferation of AI facilitated documentation for radiology reports and emphasizes that now, more than ever, quality and variability of documentation should be treated as a quality/safety measure unto itself. However, these issues are not unique to the generation, and use of, AI facilitated radiology reports, but rather can broadly be applied to multiple forms of AI facilitated documentation in general in healthcare. These issues can be broken down into a number of thematic areas, which are essential to consider in both the implementation AND evaluation of these technological disruptors.

First, and perhaps most foundational, is that most issues with technology adoption in healthcare are sociotechnical in nature. This was highlighted during the early days of EHR adoption and codified in the 8 dimensions for safe and effective EHR adoption; Hardware and Software infrastructure, People, Human-Computer Interface, Clinical Content, Workflow and Communication, Internal Orginasational Policies and Procedures, External Rules, Regulations and Pressures, and System Measurement and Monitoring. ⁵ These are never more important than in the age of AI integration, where the degree of automation can magnify many of these domains. This article by Toker and colleagues highlights the importance of this, with the lack of pre-existing best practices for reporting of radiology results, and the lack of infrastructure to ensure adherence to any established standard. ⁴ The concern is this now has the potential to magnify errors and miscommunication when AI is layered on top. The goal of AI is to help make a well-run infrastructure more efficient and better, but if the underlying practice is variable and suboptimal, the AI will fail to live up to expectations. Thus, any implementation of AI facilitated documentation tools should include an evaluation of the underlying sociotechnical environment onto which it will be layered. However, the unprecedented need for tools to combat provider documentation burden and burnout, combined with the rapid evolution of the technology, the lack of regulations and the black-box nature of many of the underlying algorithms, has resulted in deployment far outpacing structred evaluation as described above.

Second, is to understand the comparator for judging the quality of AI generated output. This needs to encompass 2 domains. First, the gold standard to which the AI generated output is being compared needs to well described and justified. Simulation, by allowing for recreation of the entire sociotechnical environment and control of clinical content in a way which does not compromise patient safety or privacy, provides an invaluable tool to help establish a true gold standard for evaluating AI generated output. In the case of AI scribes, simulation has been used to demonstrate a significant number of errors of omission and commission, and high variability between systems. ^{6 7 8} These errors not only occur in multiple phases of note generation, but can have profound impacts on cognitive reasoning, both for the author and the consumer.⁹ While this type of testing is important to allow for benchmarking, both within a system (as underlying models evolve), and between systems (in case of purchasing etc…), it must also be acknowledged that current human generated output is also flawed with errors documented both on objective evaluation and through patient reports. ^{7 10 11} Thus, we must always be cautious of holding AI to a standard greater than we hold our current workflows, but rather realize that AI gives the opportunity to reach a higher level of consistency and accuracy and fundamentally improve patient safety, but only, if both gold standard testing combined with optimization of output and workflow is done.

Third is to ensure that evaluation of AI generated output includes all of the ways the output can impact patient care. Currently, most forms of AI-facilitated documentation have been developed and promoted as tools to improve provider well-being, reduce documentation burden and increase operational efficiency. However, this evaluative focus overlooks the downstream uses of the generated content. Providers acknowledge and recognize that there are multiple competing uses of documentation, from supporting billing andquality measures, to clinical decision making and diagnostic excellence. ¹² While all of these downstream uses of documentation should be assessed, there is a paucity of information of how AI generated output facilitates communication and transmission of information to subsequent, downstream consumers of the information. At a most basic level, studies document that AI generated notes are longer that human generated ones and have similar degrees of readability.¹³ However, beyond this, it remains to be determined whether these notes are read at differential frequencies by providers and/or patients, and how they impact clinical decision making. The structured framework presented by Toker and colleagues is a critical first step for radiology reporting. ⁴ However, it will be important for this to be expanded to other forms of assisted documentation and information gathering, to ensure there is an ability to facilitate structured evaluation of the impact on these tools on downstream consumers. This to not only ensure safety, but to also serve as a baseline for iterative redesign and optimization of the output.

However, the newest and, likely most influential, consumer of AI generated output, will be other AI facilitated documentation tools. There is a recent growth, both through independent vendors and from EHR vendors themselves, to use LLMs to summarize the medical record, to either facilitate chart review or create de novo summarization (e.g. discharge summaries). ^14–16 However, this introduces another set of considerations for evaluation of both safety and impact. First, if summarized notes contain errors (either Human or AI generated) these AI summarization tools will serve as a force multiplier for inaccuracies already extant in the medical record and magnify cognitive bias. In essence, this risks the creation of a new form of “chart lore” and even fundamentally altering the actual content of the medical record. One study suggests that as the medical record is populated with more and more AI content, this will lead to a recursive degeneration of diversity of medical lexicon and diagnosis over time. ¹⁷ Second, as with other forms of AI assisted documentation, these outputs have variable quality and accuracy when compared to either human or gold-standard output, with patient characteristics likely an important confounder. ^{16 18} Third, is the concern that all relevant information may not be included in the summarization itself (another form of omission), this impacts both summarization tools and primary AI assisted documentation which can selectively “carry-forward” information from prior documentation. ⁹ Finally, and of even greater importance from a quality and safety perspective, is the variability in what data are used in these summarizations. Many initial studies only summarize narrative text, as opposed to including other structured data elements (e.g. lab values, orders) and thus these tools are merely summarizing other notes, as opposed to directly extracting structured data. Thus, any cognitive blind spots in documentation are carried forward and codified in these summarization tools. As with primary AI facilitated documentation, while AI generated summarization may be equivalent to flawed, human generated output, evaluation to a gold standard is necessary if the goal is to fundamentally improve safety and care quality. Beyond this, it is critical that there is transparency from vendors/developers on what data are and are not included in these summarizations so primary users of the tools can effectively judge the quality of the output. ⁹ Central to this is a standardized framework for display of output and attribution, in manner similar to what is proposed by Toker and colleagueset al for creation of AI generated radiology reports. ⁴

Finally, all of this must be contextualized with who, where and when these tools are used. Unlike AI powered algorithms which are embedded in Clinical Decision Support in EHRs, and thus run in the background, AI assisted documentation is designed with the purpose to improve documentation burden and is thus an “opt-in” approach. Initial data with AI powered scribes suggest a bimodal distribution of adoption, with high and low users. ^{19 20} Further, even within high adopters, AI scribes are used less than half the time, on average. While some factors, such as visit type and specialty explain some of this variance, other factors such patient complexity, patient language and provider perception and trust in the technology are likely important confounders and have yet to be fully explored ²⁰. This makes identification of when AI assisted documentation is employed a critical component of monitoring and understanding the impact of these tools on quality and safety.

In conclusion, as AI facilitated documentation continues to increase in prevalence and complexity, it will be critical to have a strong foundation to guide the nature, structure and content of the output, to assess both the direct quality, but the overall impact on patient, provider and health system outcomes. Doing so, will truly allow us to go where no health system has gone before; a state of zero preventable harm.