Generative artificial intelligence for inpatient documentation summarization: mixed-methods quality assessment and early real-world experience

Abstract

Objective

We evaluated the quality and adoption of a large language model (LLM)-based summarization tool for ongoing hospital care.

Materials and methods

An AI summarization tool was created to provide a “Patient Story”, specialty-specific “Recent Notes”, and “Recent Events” over the previous 24 or 72 hours. We conducted a pragmatic mixed-methods quality assessment at three tertiary-care academic hospitals utilizing (1) Provider Documentation Summarization Quality Instrument (PDSQI-9), (2) utilization analytics, and (3) qualitative end user feedback. The PDSQI-9 included whether summaries were accurate, cited, comprehensible, organized, succinct, non-stigmatizing, synthesized, thorough, and useful. 512 users were given access, from whom 52 participants submitted 205 surveys (10.2% response rate).

Results

52 respondents submitted an average of 4.3 surveys (range 1–8). Users rated the tool favorably across all PDSQI-9 domains, with a combined average score of 4.68 (range 4.56–4.78, S.D. 0.67) across the eight domains scored on a 5-point modified Likert scale. Utilization metrics demonstrated strong uptake with frequent views. Positive qualitative feedback revealed cognitive offloading for complex patients and effective summarization of medical problems. Critical feedback showed a need to cross-reference narrative notes to current-state data and lack of detailed specialty-specific summarization.

Discussion

Generative artificial intelligence has emerged as a potentially transformative technology for generating succinct, verifiable summaries of ongoing care. In this pragmatic implementation, end-users indicated high perceived quality across PDSQI-9 domains.

Conclusions

An LLM-based summarization tool for ongoing hospitalization care was rated of high quality by diverse clinicians in real-world settings, demonstrated a favorable safety profile, and showed sustained utilization.

Background and significance

Inpatient medical practitioners must process an increasing volume of information within the electronic health record (EHR), which constitutes a substantial portion of their daily time and cognitive burden for patient care.¹ Clinicians are expected to maintain awareness of complex patient histories, initial presentations, progress notes, specialty consult notes, new diagnostic results, medication changes, nutritional updates, care management, and more for each of their patients. Generative Artificial Intelligence (AI) leveraging Large Language Models (LLMs) promises to process this myriad of clinical data and has now advanced beyond experimental demonstrations to become integrated into major EHR platforms. However, in-depth quality evaluation frameworks of these tools are early in development.^2,3

Early applications in medical practice have included chart summarization, clinical hand-offs, creation of hospital dismissal summaries, and generation of In-Basket responses to patient questions.^4–10 A growing body of evidence suggests that these LLM summarizations can produce clinical summaries that are comparable to, or in some domains surpass, traditional written summaries.^6,11,12 With new tools becoming available, testing for accuracy and clinical benefit will continue to be critical.¹³

Recently, to assist in this evaluation, the Provider Documentation Summarization Quality Instrument (PDSQI-9) was developed and validated for the task of multi-document clinical summarization.¹⁴ The nine attributes included in this instrument are whether a summary is accurate, cited, comprehensible, organized, succinct, non-stigmatizing, synthesized, thorough, and useful. However, the PDSQI-9 does not necessarily address specific nuanced feedback from clinicians, nor does it address end-user adoption. We thus sought to apply the PDSQI-9 instrument alongside both qualitative end-user feedback and adoption trends to assess a novel AI summarization tool called “Inpatient Insights” (Epic Systems Corporation ^©).

Inpatient Insights was developed with input from our organization as an EHR-integrated generative AI feature to analyze recent clinical data and summarize important updates using an LLM for clinicians. The base LLM model for Inpatient Insights is ChatGPT (version 4, Open AI ^©). The tool provides four separate LLM summaries that can be generated for (1) the “Patient Story,” (2) “Recent Notes” filed by subspecialty, (3) “Recent Events” in the past 24-hours and (4) “Recent Events” in the past 72-hours. The specific prompting configuration is proprietary, but our clinicians assisted during the process of software development with early testing to ensure the design of this tool addressed clinical needs and was tailored to relevant clinical workflows.

Objective

We sought to evaluate the quality of a novel AI summarization tool for ongoing hospital care in real-world clinical settings at multiple tertiary academic hospital sites within a large integrated healthcare system. As indicated above, there have been numerous studies assessing LLM chart extraction, but we felt the need to address a gap in literature regarding the specific application of real time chart summaries in the care of complex patients. To do this, the AI features of this tool were evaluated using the PDSQI-9 framework, qualitative end user feedback, and utilization metrics.

Materials and methods

The study was designed as a mixed methods quality assessment of a novel EHR-embedded AI summarization tool. Evaluation was conducted at three academic tertiary hospital sites that utilize a single integrated electronic health record (Epic Systems Corporation ^©). All data was stored and used internally by Mayo Clinic. A waiver of consent was granted by the Institutional Review Board.

Between August and October 2025, 512 pilot users, including physicians, advanced practice providers, respiratory therapists, physical therapists, occupational therapists and pharmacists, were given access to the Inpatient Insights tool. The patient populations served by these clinicians included intensive care units, general medical, general surgical and specialty units, such as organ transplantation. The units were chosen to ensure the inclusion of complex medical and surgical patients. All clinicians in the selected units were given access to the tools. Providers were given instructions to refer to source notes directly, verify information, consider omitted information, and avoid copying and pasting information from the generated summaries.

The Inpatient Insights tool was based on ChatGPT version 4, and proprietary prompts generate four summary types: a “Patient Story” with background history, events during admission and recent events; a “Recent Notes” organized by specialty; and “Recent Events” summaries for either 24 or 72 hours. Regarding the subtypes of summarization and typical use cases, the “Patient Story” feature provides the highest level overview, including the background, reason for admission and recent events. A common use case is the review of a new patient for a broad overview of their hospitalization thus far. The “Notes Summary” meanwhile is particularly useful for subspecialty review—for example, focusing on nutrition, cardiology, or infectious disease issues. The “Recent Events” summaries provide more detail regarding either the previous 24 hours—which typically might be used for review of updates since the previous day—or 72 hours, which can facilitate rapid review of a known patient whom the clinician has not seen over the weekend. While it was not an explicit outcome of this study, survey responses favored review of “Patient Story” foremost, followed by 24-hour events, “Recent Notes,” and 72-hour events.

Information sources included: Emergency Department notes; history and physical, progress and consult notes; vital signs, intake and output data; problem list; medications and orders; results including laboratory, microbiology and imaging. Providers could use any combination of these summaries and generate summaries on demand during any point in their desired real-world clinical workflow. Figure 1 shows an example of the Patient Story extraction and summary.

Figure 1.

Sample patient story summary (© 2026 Epic Systems Corporation, with permission).

We assessed the quality of the summaries using the PDSQI-9 instrument (see Supplemental Material, available as supplementary data at Journal of the American Medical Informatics Association online).¹⁴ Surveys contained a modified 5-point Likert scale (1- not at all to 5-extremely) and included responses for nine domains: accuracy, citations, comprehensibility, organization, succinctness, absence of stigmatization (a yes/no response only), synthesis, thoroughness and usefulness. Qualitative data was obtained via free text comments from providers included in the same survey form. Surveys were collected using the Research Electronic Data Capture (REDCap) tool. The participants were instructed to complete one survey for each patient for whom summarized information was viewed, in real time after completing the review. Usage data, specifically view count of the AI tool, was obtained from the EHR record log. Users needed to specifically navigate to the tool for a view count to be recorded. Users were only expected to consider using the AI feature while on clinical service duties. Results were analyzed with descriptive statistics using R (version 4.5.2; The R Project for Statistical Computing).

Results

We received 205 survey responses from 52 participants (average 4.3 responses, range 1-8), each survey representing a review of one patient. Of these, 180 responses included use of Patient Story, 102 included the use of Recent Notes, 118 included the use of 24-hour Events, and 86 included the use of 72-hour events.

Results of mean PDSQI-9 scores are summarized in Table 1 and Figure 2. Across PDSQI-9 domains, mean ratings were favorable, suggesting good perceived quality from end users. All PDSQI-9 domains mean scores across were greater than 4.5. Stigmatizing language, reflecting clinician bias, may range from defining a patient by disease (“diabetic”) to judgement of compliance (“demanding, difficult, refuses”) to more directly pejorative terms (“addict, malingerer, junkie), and can impact patient care.¹⁵ For the stigmatizing language domain, scored as yes or no, only 1.5% of responses were positive, and on review was felt to represent a term in the clinical notes that was perpetuated. We did not identify any introduction of new stigmatizing language.

Table 1.

PDSQI-9 scores for generative AI chart summarizations.

Domain assessed	Mean	Standard Deviation
Accuracy	4.68	0.67
Citation	4.73	0.57
Comprehensibility	4.78	0.52
Organization	4.72	0.61
Succinct	4.56	0.87
Synthesized	4.67	0.58
Thorough	4.66	0.78
Useful	4.60	0.73
Stigmatizing (in source notes, yes/no)	1.5%
Stigmatizing (in AI summary, yes/no)	1.5%

Figure 2.

PDSQI-9 distribution of scores for those domains on 5-point scale.

Representative positive and negative feedback is shown in Table 2. Positive themes included appreciation of rapid cognitive offloading for new or complex patients, effective summarizations of hospital problems, and highlighting potentially missed clinical details. Critical feedback was most notable for critique of how the tools appear to prioritize existing clinical note free text without sufficiently incorporating recent objective data. For example, cross-referencing active orders, labs, vital signs or imaging would have provided an LLM summary with a more updated or accurate statement. There were also instances of the AI summarization failing to adequately emphasize nuanced aspects of care, such as family dynamics. There was occasional mischaracterization of timelines (e.g, stating a patient developed a condition in the hospital that existed prior to admission). Clinicians also pointed out discrepancies or conflicting information within the source hospital notes that may have led to less accurate summarizations. There were no reports of frank falsehoods, fabrications or “hallucinations.”

Table 2.

User feedback comments regarding generative AI summaries.

Positive Feedback

Wonderful tool to review ICU transfers.

The summary captured the essence of her problem, which is polypharmacy.

Remarkably complex history, well summarized including current plans and next steps.

It identified a positive culture from 2 weeks earlier that I would have missed otherwise.

Extremely useful tool in the intensive care unit.

I like that the summary collected the admitting resident’s recommendation for ongoing goals of care discussions.

Loving using insights! Such a great tool.

Much improved! I like how it now reads in chronological order. The notes summary has also been optimized in a way that makes it much easier to understand the patient’s condition from the point of each service.

Critical Feedback

The biggest thing missing is the family’s perspective, which is currently dominating his hospital stay: they are very focused on his needing a G-tube, despite not a great deal of evidence that it will help. I’m not sure how to get AI to take that into account, but it’s crucial to understanding what’s going on at his bedside at present.

It would be more beneficial to utilize the Insights if these were based on summarizing actual lab and micro data versus summarizing what is documented predominantly in notes as it seems to lag behind for changes made with the patient that have yet to be reflected in more recent notes.

I did feel like the summary underplayed the impact of her new steroids on her planned surgery, but that is a very picky point.

Did not develop malnutrition but “found to have” [malnutrition] per note.

“Diabetes management stable on Lantus 30 units nightly and NovoLog 10 units with meals plus correctional sliding scale; HbA1c remains elevated at 10.5%. “ doesn’t make any sense [for inpatient setting]

Missing that patient had a lobectomy, this could be because the surgery note was not in, but the encounter of the procedure was there. This patient is [hospitalized for 3 weeks] so perhaps a reason for limited pulling of info.

It would be nice if this tool could take data from the Labs tab as well. Summary stated that plan was to re-draw blood cultures. Source note is from 3 days ago stating plan is to re-draw cultures. Cultures were drawn 3 days ago and are pending. It would be nice if the tool could read the labs tab and state “Cultures were drawn on 9/16 and are pending.”

Usage data are plotted in Figure 3, including daily user count and daily AI feature views. The views reflect actual engagement with the summary and do not include automatic or system-level generation. Daily user counts and AI feature views were sustained during the pilot phase, with a rising trend exceeding 1000 AI feature views per day among the total of 512 activated pilot users.

Figure 3.

User count and trend of views of AI-generated summaries.

Discussion

In this early pragmatic implementation and mixed-method evaluation of a novel inpatient AI summarization tool, end-user assessments indicated high perceived quality across PDSQI-9 domains in parallel with strong demonstrated real-world uptake reflected by sustained daily feature views. This tool uses an LLM to help address the work necessary for chart review and complex daily care delivery in inpatient and ICU settings by providing a summary of (i) the high-level “Patient Story,” (ii) specialty-organized “Recent Notes,” and (iii) “Recent Events” summaries for either 24- or 72-hour timescales. Qualitative feedback showed appreciation of how the tool provides rapid cognitive offloading for new or complex patients, effectively summarizes hospital problems, and highlights potentially missed clinical details.

Specifically, the summarization tool performed well in PDSQI-9 domains of clinician perceived accuracy, comprehensibility, organization, succinctness, synthesization, thoroughness, utility, usage of citations, and the avoidance of stigma. These findings build on prior work at our institution and others demonstrating LLM-generated summaries of hospital courses can provide high quality, consistent and clinically useful syntheses, which at times exceed human performance. ^6,12 This study adds to the literature by demonstrating a novel AI tool in real-world use earlier in the hospitalization, at which time care is ongoing and the clinical narrative is evolving. This is particularly valuable when documentation is fragmented across disciplines with complex clinical circumstances and many contributing note authors. Whereas hospital discharge summaries are finalized after the clinical course has fully taken place, this tool provides near real-time summarization, supporting ongoing clinical cognition.

At the same time, weaknesses of this AI tool in its current form were identified. Qualitative evaluation demonstrated that the LLM summarization occasionally failed to prioritize objective data over human written data in existing clinical notes. Subtle aspects important to patient care, such as family dynamics, likely would not be captured in clinical documentation. It also occasionally failed to provide sufficiently specific summaries for specialists or comprehensive data surrounding precise clinical problems. However, there were no noted reports of frank hallucinations. Errors that were found tended to be perpetuated inaccuracies in existing documentation, or in one case, perceived mild stigma that was already present in clinician notes.

This evaluation benefited from a large number of diverse users in real-world settings, complex hospitalized patients with ongoing care, and structured, real-time feedback. This study is important because generative AI for chart extraction is expanding rapidly in healthcare settings, raising some concerns regarding accuracy, quality and potential for patient harm.¹³

We found that a particular strength of this tool is that summaries generated provide traceable citations to original documentation, enabling verification. With citation capability built in, we emphasized the importance of verifying summarizations before basing clinical decisions on any information provided. This allows for a clinician-in-the-loop to safeguard care decisions. The PDSQI-9 framework highlights the importance of this, as it includes the use of citations as an evaluation domain to assess the quality of an AI clinical summary.¹⁴

Important capabilities were identified for the future development of inpatient AI summarization tools for ongoing care. Foremost, the integration of current discrete laboratory and active orders data should be prioritized over past notes data. For example, an AI summary referencing an antibiotic that was in prior notes should be cross referenced to the current active antibiotic order for clarification and accuracy. Second, specialty-specific and problem-based summarizations should be pursued for development. These should include dynamic templates that extract the most relevant data specifically for that specialty or problem. For example, in a patient hospitalized for COPD exacerbation, a summary may include (1) the most recent pulmonary consultation plan, (2) the patient’s latest PFT results, (3) number of known hospitalizations in the past year, (4) current outpatient controller and rescue regimen, (5) key relevant laboratory data, (6) latest inpatient vitals, (7) oxygen devices used, and (8) active inpatient respiratory medications. This recommendation for specialty-specific views is supported by calls for improved representation of non-general specialties in clinical generative AI development.¹⁶ Third, tools should incorporate improved temporal reasoning to mitigate timeline errors, such as assuming an event occurred during hospitalization instead of being present on admission. Fourth, automated quality evaluation checks should be built into systems for ongoing monitoring. For example, an LLM-as-a-judge approach utilizing the PDSQI-9 scale can automate evaluations with less intensive human effort.¹⁷

The study has several important limitations. Most notably, there was a low survey response rate. This carries concerns regarding sampling and response bias of enthusiastic early adopters. We also utilized self-report for safety outcomes by clinicians, which does not exclude the presence of critical errors or omissions that may have gone undetected or unreported. Next, while multiple sites and a heterogenous user base give the study strength, we did not have survey responses associated by role, which limited ability to perform role- or unit-specific subgroup analyses. Additionally, while utilization itself provided a measure of perceived value, we did not include direct measurements of efficiency, cognitive burden, or clinical outcomes. Another limitation is that because the generative AI prompts used for this tool under evaluation are proprietary, we have less sightline into its drawbacks or how to rapidly address concerns that arise. Similarly, when model versions used as the base LLM for a tool such as this change, there are unknown effects of how this may affect the quality of the output going forward.

We also suggest that the PDSQI-9 scale should not be treated as a stand-alone quality evaluation tool for real-world clinical implementations. With the high performance of modern LLMs, many clinical summaries may score well on average scores across PDSQI-9 domains, as was the case in this study. As a consequence, the criticality of errors present may not be fully captured by a mean PDSQI-9 score or distribution of scores. For example, if an LLM functions well the majority of the time, but in one out of 50 cases makes a critical error, it could still achieve good PDSQI-9 performance while being deemed unacceptable for implementation at a large organization with thousands of clinical staff. Thus, qualitative review and robust error reporting with root cause analysis must be coupled with evaluation frameworks such as the PDQSI-9. Furthermore, much of the debate regarding generative AI revolves around the value statement in terms of whether a tool is worth financial investment by an organization. For this, the PDSQI-9 may not offer insight; in comparison, basic utilization and adoption metrics may be preferred, which should be key aspects of evaluating clinical AI tools. Further augmenting utilization metrics with established evaluative frameworks for efficiency such as EHR Signal data (^© 2026 Epic Systems Corporation), NASA task load index or clinician burnout scale should also be considered.^18,19

Finally, while the scores for synthesis of information were relatively high, the current state of chart summarization by generative AI focuses on extracting key information from large amounts of data, rather than drawing inferences, conclusions or suggesting next steps in care. Indeed, the latter capabilities would be likely to draw regulatory interest by the Food and Drug Administration (FDA), under guidance regarding clinical decision support using software as a medical device.²⁰

Conclusion

Ultimately, this mixed-methods quality assessment demonstrates that an inpatient AI summarization tool achieved high reliability across PDSQI-9 domains and was associated with strong real-world uptake. Prospective studies regarding task load and clinical outcomes should be pursued to further define real-world value. We recommend clinicians continue to remain vigilant and review any AI output prior to incorporating into care. While acknowledging inherent risks in generative AI summaries, and the need for continued refinement and improvements in performance, these tools should be considered for cautious adoption in this new era of medicine.

Author contributions

Steve G. Peters (Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Validation, Visualization, Writing—original draft, Writing—review & editing), Jens P Boyum (Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Validation, Visualization, Writing—original draft, Writing—review & editing), Sean R Legler (Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Validation, Visualization, Writing—original draft, Writing—review & editing), Kate J Heise (Conceptualization, Investigation, Methodology, Validation, Writing—original draft, Writing—review & editing), and Angie Griffin (Conceptualization, Data curation, Investigation, Methodology, Project administration, Resources, Writing—original draft, Writing—review & editing), Heather Heaton (Conceptualization, Formal analysis, Investigation, Methodology, Project administration, Writing—original draft, Writing—review & editing).

Supplementary material

Supplementary material is available at Journal of the American Medical Informatics Association online.

Funding

This research was supported by the Mayo Clinic. The authors received no additional funding.

Conflicts of interest

None declared.

Data availability

Survey data are included in the manuscript. Identifiable data are restricted and not available.