Table 2. Physician perceptions of LLM-generated summaries: strengths and limitations.

Themes and subthemes from the qualitative analysis, reflecting what physicians appreciated and found lacking in the summaries, along with illustrative quotes for each subtheme.

Strengths of the summaries identified by the respondents with quotes

Limitations of the summaries identified by the respondents with quotes

1. References    “I liked the fact that there were references for verification”    “However, the clinician still has to dig into the EMR to find out details like what type of arrhythmia, but it’s helpful to have the reference from the summary.” 2. Organized    “Love how pertinent data is parsed out into categories/subheadings”    “They were succinct and contain all the information in one place.”    “pulls all relevant info and arranges it in a manner that is easy to find.” 3. Useful/High yield    “The “medication narrative and changes” section is fantastic and very high yield.”    “Most of the facts that were included were quite helpful (e.g. the Cr, types of interventions). I would like to use this in real life as a great first pass on HF or to discuss with a consultant/team”    “For those that had the NYHA class, that was useful. I appreciated the medication narrative at the bottom. History of interventions was useful.” 4. Convenient/Quick    “Easier to find that having to go through EHR.”    “It will provide reassurance and a quicker way to understand some information about the patient. It highlights key things one needs to know about a heart failure patient when they walk into the door.”    “Easy to find what should be structured data but is not inputted that way in notes (ie dry weight, history of arrhythmia) to answer simple questions” 5. Contextual information was present    “There are some narratives that gives context and not just one-word answers”    “provides a narrative for explanation re: med changes. Summarizes important studies/procedures.”

1. Better familiarity with the EHR    “Summaries were also a bit long, so I sometimes went back to the EHR (which I was more familiar with) to find the data I needed”    “At times, I had to jump around to find the information I was looking for, but this could have just been related to getting used to the layout.” 2. Organization/Layout    “The organization of the summary is somewhat counterintuitive in a way that fragments my cognitive workflow a bit. My natural thought process is type/cause of heart failure f/b the most pertinent clinical data for a rapid prognostic overview of the patient’s clinical picture -- most pertinently: coronary interventions, h/o valvular dz/arrhythmias, PPM/ICD presence, meds (with recent changes), THEN the data that i can look at if i need more information/data that is more easily found in the EHR (ie proBNPs, Cr, dry weight, etc)”    “I wonder if structuring with how we structure our assessment and plan for CHF might be useful. Something like Preload, Afterload, Contractility, GDMT or something similar. “ 3. Lack context    “cardiac med narrative is a little confusing - would be nice to separate into IP and OP events (or to set expectations for what is in a narrative - most recent hospitalization?, purpose/scope unclear, timing also very unclear.)”    “in real life - comorbidities may be more relevant (e.g. if they have bleeding issues etc.) or they may have multiple medical problems besides cardiac that complicate their management - the scope was fine for this research but in real life people may want more” 4. Accuracy    “I was primarily concerned with accuracy of information. There was one summary that was completely wrong in terms of the factual data contained. This finding made me concerned about trusting the summary in general.”    “At times there were items that were not accurate and needed confirmation. It would be good to know what is confirmed and by whom.” 5. Missingness    “all the fields that just said “NA” (such as for type of heart failure when the patient did actually have some HF). Then I stopped using the summary as much for that patient and just relied on the EMR”    “the narratives leave out some meds; sotimes [sic] med list includes more than one med of the same type (confusing). it doesn’t include date med was last dispensed.”    “Sometimes it did not contain all the info I would have wanted” 6. Redundant    “Unorganized Occasionally redundant”