Table 1.

A sampling of clinical summarization applications, organized by publication date

	Summarization approach	Input	Output	Evaluation	Deployed (when is it generated)	General Notes
NUCRSS22,26	Extraction of clinical variables, indicative	Real structured EHR data	An eight page summary of: Problem list, Vital signs, Cardiac-pulmonary-renal diagnoses, Treatments, Routine specialized laboratory examination, Suggestions to physicians regarding patient care	Laboratory study with medical students and physicians showed significant time savings and increased accuracy Randomized controlled trial found showed that the NUCRSS improved process level (patient’s length of stay and increased the amount of laboratory tests ordered) outcomes and may have improved care.	Yes (each patient visit)	Early example of a summarizer One of the few summary evaluations that demonstrate an impact on quality of care and process outcomes.
STOR27	Extraction of clinical variables, indicative	Real structured and unstructured EHR data	Loosely customizable, summary which included both time- and problem- oriented views	Clinical study found that clinicians were better able to predict their patient’s future symptoms and laboratory test results when the using medical record in addition to STOR as opposed to just the medical record.	Yes (each patient visit)	Early example of a summarizer One of few examples of task-based evaluation The summary is context-dependent on the patient, but the context is manually determined by the clinician (what problems are active, what observations are relevant, etc.)
Powsner and Tufte11,28	Extraction of psychiatric variables and recent notes, indicative	Simulated structured, unstructured and genealogy data	A one-page summary that visualizes the most salient content (as defined by recency) of the patient record.	None	No	A widely referenced prototype that continues to serve as a model for current EHR visualization and summarization applications.
Lifelines29,30	Extraction of clinical variables, indicative	Simulated structured data	Holistic interactive patient summaries using a temporal data view on top of the raw EHR data. Displays facts as lines on graphic time axis according to their temporal location and categories/significance are represented by color and thickness.	The original Lifelines application was evaluated for work with juvenile youth records29 by a small group of users who reported enthusiasm but mentioned potential biasing by the system’s graphics.	No	Lifelines is probably the most well-known summarizer tool. The display has served as a model for future timeline-view clinical summarizers Lifelines2 was created for research and examining many patients together.
CliniViewer23	Extraction of concepts from text, indicative	Real unstructured EHR data	Combined NLP techniques and presented a tree view of a patient’s problems extracted from the narrative text to the clinician. Displays concepts in context when clicked.	The system was able evaluated on accuracy and speed using real discharge summaries but no evaluation with clinicians was conducted.	No	One of the first examples of summaries created using NLP Allows for customizable user views Works on top of the MedLEE31 NLP engine which handles modifiers
IHC Patient Worksheet32	Extraction of clinical variables, indicative	Real structured EHR data	1–2 page outpatient summary of: Demographics, Problems, Medications, Laboratory tests, Actionable advisories	A retrospective cohort study found that compliance with HbA1c testing was higher for patients who had a worksheet printed than for those who did not.	Yes (each patient visit)	One of the few example of a clinical outcome tested in the evaluation
CLEF33–35	Abstraction from text and extraction of clinical variables, indicative	Simulated structured and unstructured cancer patient data.	An interactive display of both navigational capabilities for the EHR (indicative) and generates textual summaries (abstractive) to enhance comprehension. It uses information extraction techniques to identify classes of data and relationships between them.	None	No	One of the few natural language generation systems created for medical histories. Represents histories as a semantic network of events organized temporally and semantically. Lists requirements that are very relevant to general designers of clinical summaries – the list was generated via initial requirements elicitation process. Uses a logical model of cancer history

	Summarization approach	Input	Output	Evaluation	Deployed (when is it generated)	General Notes
KNAVE-II36	Abstraction and extraction of clinical variables, informative	Real structured data on bone marrow transplant patients	Interactive data display of abstracted and raw protocol-based care data containing a tree-browser and time chart.	A crossover study compared KNAVE-II with paper charts and Excel spreadsheet. Users produced quicker answers, had somewhat better accuracy and preferred KNAVE-II however it did not achieve a very high system usability score.	No	Performs semantic, temporal, and context abstraction. Requires domain-specific ontologies. Consists of a knowledge base, abstraction generator, navigation engine, and visualization. Lists 12 desiderata for interactive, time-oriented clinical data that should be used to guide future summarization work as well.
BabyTalk (BT-45)37,38	Abstraction of ICU data streams, informative	Real raw neonatal ICU data streams	Automatically generated natural language to describe ICU data streams for easier comprehension by the nursing staff.	A laboratory study found that human-generated text summaries of ICU streams helped nurses predict their patient’s trajectories’ better. The team is working to create automatically generated text summaries that perform as well as human-generated summaries.	No	A novel example of summarizing graphical ICU information by generating text.
Were et al.39	Extraction of clinical variables, indicative	Real structured EHR data from OpenMRS	Patient summary for use in an HIV clinic in Uganda	A pre–post study design using time-motion study techniques and surveys. The authors found that providers who used the summary sheet were both able to spend more time directly with their patients and the average length of visit was reduced by 11.5 min.	Yes (each patient visit)	A largely successful process outcome. Explores the utility of summaries in a low-resource setting.
TimeLine/AdaptEHR40,41	Abstraction from text and extraction of clinical variables, informative	Real structured, unstructured and image data on brain tumor patients	An interactive data display that summarizes and integrates various pieces of the EHR including images and free text.	A pilot study on Timeline found that although the initial learning curve was high, with time, the clinicians were able to perform image review quicker and were more confident in their clinical conclusions than when they used the EHR display.	No	Timeline had manually coded rules while AdaptEHR aims to automatically infer rules and relationships from ontologies and graphical models, the publication states that the conditional probability tables are not yet defined. Has four dimensions of representing data: time, space (where physical location of tumor), existence (certainty), and causality (treatment response treatment)
HARVEST42	Extraction of concepts from text and clinical variables, indicative	Real structured and unstructured EHR data	A problem-based, interactive, temporal visualization of a longitudinal patient record.	A task-based, timed evaluation found no difference in ability to extract, compare, synthesize and recall clinical information when using HARVEST in addition to the EHR, when carried out with subjects who had no prior experience with the summarization tool.	Yes (Real time)	Aggregates information from multiple care settings Operates on top of a commercial EHR system using HL7 messages Distributed computing infrastructure to enable real-time summarization.

The inputs, outputs, methods, and evaluation strategies are listed along with notable additional information for each summarizer.