The promises and limitations of artificial intelligence for quality improvement, patient safety, and research in hospital medicine

INTRODUCTION

The emergence of large language models (LLMs) with user-friendly chatbot interfaces has revolutionized access to advanced artificial intelligence (AI). Tools previously gated behind innumerable access barriers are now widely available. However, as more members of the public explored the potential applications of these models, the simplicity of the interfaces belied the many nuances of the underlying technology, resulting in several high-profile mishaps when using LLMs.¹ This increase in public awareness was accompanied by wider scrutiny of the potential transformative impact of AI on society in general, both good and bad, resulting in an executive order on the responsible use of AI.²

Given that the consequences of errors in healthcare are potentially life-altering, a thoughtful, measured approach is imperative for the introduction of new technologies. While the focus is frequently on improving clinician performance, the potential for novel use cases in quality improvement and research also deserves rigorous exploration. As a key invested party within these areas, the challenge for hospital medicine lies in realizing the promise of these AI-enabled or assisted innovations while mitigating and overcoming their limitations (Table 1).

TABLE 1.

Thematic summary.

	Quality improvement	Research
Promises	Standardized and personalized clinical decision support Real-time monitoring	On-demand mentorship Automated chart review Streamlined clinical trials
Limitations	Nongeneralizable models Technical limitations of the EHR Health system implementation barriers	Lack of gold standards Lack of transparency

Abbreviation: EHR, electronic health record.

PROMISES OF AI IN QUALITY IMPROVEMENT AND PATIENT SAFETY

The incorporation of AI into clinical medicine is frequently envisioned as the creation of advanced clinical decision support tools that aid healthcare providers at the point of care. These tools can be categorized into two distinct types, each with its own unique set of benefits. The first type focuses on the standardization of processes that might otherwise vary widely, such as the initial workup sent by primary care physicians when placing referrals.³ The second type enhances the personalization of care, such as the use of AI to direct the overall diagnostic approach,⁴ perform diagnostic stewardship,⁵ and guide surgical decision-making and perioperative care.⁶ These two “flavors” of AI tools enable more precise, efficient, and patient-centered approaches to healthcare, marrying the rigors of standardized treatment with the nuanced understanding of individual patient needs.

The prowess of AI extends to automated real-time patient monitoring tools. Many act through clinicians as intermediaries by flagging potentially actionable events for further review such as inappropriate antimicrobial use,⁷ arrhythmias,⁸ iatrogenic hypoglycemia,⁹ sepsis,¹⁰ impending ICU escalations,¹¹ readmission risk,¹² the accidental inclusion of confidential information in shared notes,¹³ and the need for advanced care planning.¹⁴ Alternatives to this model include fully automated titration of medications¹⁵ as well as the prediction of nonclinical outcomes such as emergency department volumes.¹⁶ The result is an increased awareness of patient status and dynamicity throughout their clinical journeys, enabling previously impossible patient safety and quality improvement interventions.

LIMITATIONS OF AI IN QUALITY IMPROVEMENT AND PATIENT SAFETY

Despite its promise, the application of AI to quality improvement and patient safety remains limited. In terms of the models alone, the promise of improved standardization frequently falls apart in the face of challenges with the generalization of algorithms across practice settings (e.g., differences between patients admitted to general medicine versus oncology inpatient services), the erosion of model accuracy in the face of changing practice patterns and data drift over time, and the potential for models to perpetuate pre-existing disparities.^17,18 The practical application of machine learning algorithms to deliver instantaneous clinical decision support at the point of care in real-time is also challenging, and currently requires ad hoc solutions to overcome technical barriers associated with immature infrastructure and standards for data exchange.¹⁹

More importantly, the success of AI technologies rests on the interaction between the models themselves, the humans they interact with,²⁰ and the health systems within which they are embedded. For instance, a model with equal predictive performance across all patients during training and validation may ultimately exacerbate inequalities in patient outcomes once deployed if not assessed and monitored following implementation.²¹ The attitude and knowledge of healthcare providers towards AI, the interpretability of AI models, and the potential for automation bias can all hinder the effective deployment of AI. Just-in-time education in the form of effective user interfaces can mitigate some of these issues but cannot substitute for increasing baseline data literacy among clinicians. Beyond the optimization of the clinician-machine interface, the broader implementation framework must account for other health systems considerations such as patient attitudes, competing priorities for clinicians, financial sustainability,²² and potential legal ramifications.²³

PROMISES OF AI IN RESEARCH

Scholarly productivity remains an area of concern within academic hospital medicine, with a minority of faculty achieving promotion to associate or full professor and a median of zero publications.²⁴ Clinical burden is commonly cited for lack of productivity, suggesting that tools that assist researchers with efficiency may be helpful. A recent survey of postdocs by Nature showed that about 1/3 of postdocs are using generative AI chatbots for research.²⁵ As a “thought partner,” AI may contribute to study design, identifying relevant variables and potential correlations that would otherwise go unnoticed. When functioning as a “coding assistant,” it can rapidly generate boilerplate code, allowing researchers to focus on higher-level analytical tasks. As a “research assistant,” AI can assist with literature searches and figure generation. As an “editor” in the writing process, AI can correct grammar, distill complex thoughts into digestible frameworks, and assist with code switching. AI's role as an informal “reviewer” can be particularly beneficial, offering feedback and prompting authors to consider different aspects of their work that might require further elucidation or support. This is especially true given the relative paucity of experienced research and scholarship mentors in many academic hospital medicine groups, which can hinder junior faculty career progression.

One of the most time-consuming tasks in clinical research is manual chart review. Automated chart reviews for the identification of relevant predictive variables and labels can significantly cut down on the associated time and resource expenditure required for clinician review.²⁶ Natural Language Processing (NLP) tools can parse unstructured data (particularly important when capturing social determinants of health, which are often scattered throughout multiple documents), while computer vision algorithms can supplement the analysis of telemetry and medical imaging²⁷ beyond what is available in written reports. Further, the development of novel natural language interfaces can help minimize analyst time and resources for cohort identification and data extraction.²⁸ Ultimately, automated data processing pipelines can result in the development of highly capable predictive clinical models.²⁹

The potential for AI to optimize clinical trials deserves special mention given the enormous associated cost. AI has the potential to revolutionize participant screening and selection by identifying eligible subjects from the EMR or other large databases.^30,31 LLMs can potentially assist with the creation of easily understandable content at various literacy levels in different languages, enabling recruitment of a more diverse and inclusive participant pool. Finally, just as AI can automate documentation and improve standardization in clinical care, it can similarly improve the efficiency of clinical trials through decreased protocol deviations and improved trial reporting.

LIMITATIONS OF AI IN RESEARCH

A fundamental challenge in training machine learning models for clinical applications is the absence of universal gold standards for clinically relevant outcomes or processes. There is wide variability in clinical practice with poor interclinician reliability when it comes to how individual clinicians diagnose and treat the same conditions. This is especially true for common diagnoses in hospital medicine, such as congestive heart failure, pneumonia, and COPD exacerbation that are defined clinically rather than through clear objective criteria. These discrepancies can pollute the training data resulting in unusable models. Addressing this challenge requires a concerted effort to foster consensus among healthcare professionals and to establish evidence-based benchmarks, accounting for the full spectrum of severity and acuity seen even within the same diagnoses in hospital medicine.

Finally, transparency is paramount when utilizing AI in research. The extent and nature of AI involvement must be explicitly stated in research methodologies to ensure the integrity of the study and to maintain the trust of the scientific community and the public. In particular, generative AI tools introduce risks such as the spread of misinformation or fabrication of citations, necessitating critical evaluation by subject matter experts. Most journals have adopted guidelines for the disclosure of AI assistance that researchers must adhere to, thereby maintaining transparency.³² Specifically with respect to authorship, the Committee on Publication Ethics (COPE) position statement bars AI tools from being credited as a coauthor as they are unable to take responsibility for the integrity of submitted work.³³ The investigators and authors remain ultimately responsible and accountable for the work presented, regardless of the tools used to do the work.

CONCLUSION

AI offers promising advancements in QI and research within hospital medicine, providing tools that may enhance decision-making and clinician efficiency, improve patient outcomes, lower healthcare costs, and increase the pace of scientific discovery. However, the adoption of these tools comes with significant challenges, particularly in terms of integration into clinical practice, demonstration of impact, and ethical considerations such as the potential to exacerbate pre-existing biases. As the rapid pace of innovation in AI continues, its relationship with clinicians and researchers must be carefully managed to ensure the best outcomes for patients and the integrity of scientific discovery.