Table 3. Physician suggestions for improving LLM-generated summaries.

Themes and subthemes capturing physician feedback on ways to enhance summary quality, supported by representative quotes for each subtheme.

Suggestions for improvement of the quality of the summaries
1. Design changes
i) Hover
“being able to hover over the footnote to see the note it’s referencing would be nice. Needing to repeatedly scroll to the bottom of the summary to see the chronological relationship between the dry weight in [3] and meds pulled from [6] is suboptimal”
ii) Layout
“Shorten [sic] them would be helpful. Rearranging them so TTE, meds were up front”
“Philosophical decision on whether you want to organize it SOAP format vs by subcategory of cardiology of HF vs arrhythmia vs CAD - CMY then interventions then TTE is less intuitive to me as a generalist”
“would rework the rearrangement of data w clearer divisions. MEDS, PROCEDURES, etc”
“Make them less redundant and more organized based on how a clinician reviews the chart.”
iii) Visual
“There are potential visual changes to the summary in terms of design I might make.”
2. Provide more context around discrepancies and inconsistencies
“Clearer identification of the dry weight. Clearer identification of last diuretic dose.”
3. Add clinical decision support
“For the question on diuretic dosing, I found myself having to remember the conversion from PO to IV. It would be great if the summary already contained this information.”
4. Plan
“A clearer and consistent plan instead of just summarized information”